1
0
mirror of https://git.FreeBSD.org/src.git synced 2024-11-21 07:15:49 +00:00
Commit Graph

288528 Commits

Author SHA1 Message Date
Konstantin Belousov
24e38af60a DMAR: add knob to disable RMRR entries installation into domains
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2023-12-26 03:28:22 +02:00
Konstantin Belousov
30ce85ca11 iommu_gas: add ddb 'show iommu_domain' command
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2023-12-26 03:28:22 +02:00
Konstantin Belousov
a869643e18 iommu_gas_match_one(): check for underflow
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2023-12-26 03:28:22 +02:00
Konstantin Belousov
7153d5e4bc dmar(9): style, fix indent
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2023-12-26 03:28:22 +02:00
Konstantin Belousov
6afa2333d2 iommu: remove leftover sys/cdefs.h includes
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2023-12-26 03:28:22 +02:00
Dimitry Andric
1ff41cad71 Minimize libc++ errno-related header diffs with upstream
In commit 88640c0e8b the new EINTEGRITY errno value was added, and
this caused us to carry a patch for upstream libc++ since that time.
Because it can cause merge conflicts when importing libc++ code from
upstream, I have submitted an upstream pull request to get most of that
patch integrated.

It turns out that we do not need the errno.h part of it at all, since
all supported FreeBSD versions define EOWNERDEAD and ENOTRECOVERABLE,
and therefore the block that juggles with ELAST values is never used in
FreeBSD. At the moment it only applies to older versions of Linux, or
possibly other platforms.

Therefore the only part that needs to stay is the definition of a enum
errc value for EINTEGRITY, and this is made optional upon EINTEGRITY
being defined, to make it suitable for upstreaming.

No functional change is intended.

MFC after:	1 week
2023-12-25 18:19:12 +01:00
Robert Clausecker
fb9c25a91f share/man/man7/simd.7: document SIMD-enhanced memrchr implementation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
2023-12-25 15:00:08 +01:00
Robert Clausecker
fb197a4f77 lib/libc/amd64/string: add memrchr() scalar, baseline implementation
The scalar implementation is fairly simplistic and only performs
slightly better than the generic C implementation. It could be
improved by using the same algorithm as for memchr, but it would
have been a lot more complicated.

The baseline implementation is similar to timingsafe_memcmp.  It's
slightly slower than memchr() due to the more complicated main
loop, but I don't think that can be significantly improved.

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42925
2023-12-25 15:00:05 +01:00
Robert Clausecker
691ff1832e lib/libc/tests/string: add memrchr unit tests
The "values" test case is specifically crafted to detect the off-by-one
error previous discovered in the scalar strchrnul implementation.

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42925
2023-12-25 14:59:58 +01:00
Robert Clausecker
e0d4f419ac lib/libc/string: document restrict qualification of memccpy() arguments
POSIX.1-2004 and the upcoming C23 agree that memccpy()'s arguments
are restrict qualified and must not overlap.  In 2002, restrict
qualifiers were added to <string.h>'s declaration of the function.
Make things official and document that the arguments must not
overlap.

See also:	61b60edfd3
Approved by:	kib
MFC after:	1 month
MFC to:		stable/14
2023-12-25 14:59:50 +01:00
Robert Clausecker
ea7b13771c lib/libc/amd64/string: implement strncat() by calling strlen(), memccpy()
This picks up the accelerated implementation of memccpy().

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D42902
2023-12-25 14:59:48 +01:00
Robert Clausecker
5fa0fbf40b share/man/man7/simd.7: document simd-enhanced memccpy, strncat
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	HTTPS://reviews.freebsd.org/D42902
2023-12-25 14:59:46 +01:00
Robert Clausecker
fc0e38a7a6 lib/libc/amd64/string: add memccpy scalar, baseline implementation
Based on the strlcpy code from D42863, this patch adds a SIMD-enhanced
implementation of memccpy for amd64. A scalar implementation calling
into memchr and memcpy to do the job is provided, too.

Please note that this code does not behave exactly the same as the C
implementation of memccpy for overlapping inputs. However, overlapping
inputs are not allowed for this function by ISO/IEC 9899:1999 and neither
has the C implementation any code to deal with the possibility. It just
proceeds byte-by-byte, which may or may not do the expected thing for
some overlaps. We do not document whether overlapping inputs are
supported in memccpy(3).

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42902
2023-12-25 14:59:42 +01:00
Robert Clausecker
e4b7b0bcbc lib/libc/tests/string: add unit tests for memccpy()
Adapted from the strlcpy() unit tests.

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
2023-12-25 14:59:40 +01:00
Robert Clausecker
6fdcb85b1c share/man/man7/simd.7: add forgotten aarch64 string functions
I previously forgot to mention these as they are set up through
contrib/arm-optimized/routines/string.

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
2023-12-25 14:59:37 +01:00
Robert Clausecker
d4a5605d9f share/man/man7/simd.7: document scalar/baseline strlcpy, strlcat
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42863
2023-12-25 14:59:35 +01:00
Robert Clausecker
2b7b03b7ae lib/libc/amd64/string: implement strlcat() through strlcpy()
This should pick up our optimised memchr(), strlen(), and strlcpy()
when strlcat() is called.

Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42863
2023-12-25 14:59:31 +01:00
Robert Clausecker
74d6cfad54 lib/libc/amd64/string: add strlcpy scalar, baseline implementation
Somewhat similar to stpncpy, but different in that we need to compute
the full source length even if the buffer is shorter than the source.

strlcat is implemented as a simple wrapper around strlcpy.  The scalar
implementation of strlcpy just calls into strlen() and memcpy() to do
the job.

Perf-wise we're very close to stpncpy.  The code is slightly slower as
it needs to carry on with finding the source string length even if the
buffer ends before the string.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D42863
2023-12-25 14:56:05 +01:00
Robert Clausecker
f7098b8659 lib/libc/tests/string: add unit test for strlcpy
A straightforward derivation from the stpncpy unit test.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42863
2023-12-25 14:56:02 +01:00
Robert Clausecker
c983825a77 share/man/man7/simd.7: document SIMD-enhanced strcat
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42600
2023-12-25 14:55:59 +01:00
Robert Clausecker
aff9143a24 lib/libc/amd64/string/strcat.S: enable use of SIMD
strcat has a bespoke scalar assembly implementation we
inherited from NetBSD.  While it performs well, it is
better to call into our SIMD implementations if any SIMD
features are available at all.  So do that and implement
strcat() by calling into strlen() and strcpy() if these
are available.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Reviison: https://reviews.freebsd.org/D42600
2023-12-25 14:55:53 +01:00
Robert Clausecker
75a9e22506 share/man/man7/simd.7: document simd-enhanced strncpy, stpncpy
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42519
2023-12-25 14:55:50 +01:00
Robert Clausecker
e19d46c808 lib/libc/amd64/string: implement strncpy() by calling stpncpy()
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42519
2023-12-25 14:55:48 +01:00
Robert Clausecker
90253d49db lib/libc/amd64/string: add stpncpy scalar, baseline implementation
This was surprisingly annoying to get right, despite being such a simple
function.  A scalar implementation is also provided, it just calls into
our optimised memchr(), memcpy(), and memset() routines to carry out its
job.

I'm quite happy with the performance.  glibc only beats us for very long
strings, likely due to the use of AVX-512.  The scalar implementation
just calls into our optimised memchr(), memcpy(), and memset() routines,
so it has a high overhead to begin with but then performs ok for the
amount of effort that went into it.  Still beats the old C code, except
for very short strings.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D42519
2023-12-25 14:55:42 +01:00
Robert Clausecker
6fa9e7d873 lib/libc/tests/string/stpncpy_test.c: extend for upcoming SSE implementation
This adds additional unit tests validating the function for
All possible alignment offsets of source and destination.

Also extend the test to allow testing of an external stpncpy
implementation, which greatly simplifies the development of
custom implementations.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42519
2023-12-25 14:55:37 +01:00
Robert Clausecker
75cb202698 share/man/man7/simd.7: document amd64 SIMD use for strsep()
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D42346
2023-12-25 14:55:34 +01:00
Robert Clausecker
fd2ecd91ae lib/libc/amd64/string: implement strsep() through strcspn()
The strsep() function is basically strcspn() with extra steps.
On amd64, we now have an optimised implementation of strcspn(),
so instead of implementing the inner loop manually, just call
into the optimised routine.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42346
2023-12-25 14:55:30 +01:00
Robert Clausecker
dd1c2e887c share/man/man7/simd.7: document strrchr scalar, baseline implementation
Also mention missing rindex() entry, which is provided through
strrchr().

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42217
2023-12-25 14:55:26 +01:00
Robert Clausecker
2ed514a220 lib/libc/amd64/string: add strrchr scalar, baseline implementation
The baseline implementation is very straightforward, while the scalar
implementation suffers from register pressure and the need to use SWAR
techniques similar to those used for strchr().

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42217
2023-12-25 14:55:22 +01:00
Robert Clausecker
f5edd8450e share/man/man7/simd.7: document strncmp amd64 scalar, baseline implementations
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42122
2023-12-25 14:55:18 +01:00
Robert Clausecker
14289e973f lib/libc/amd64/string: add strncmp scalar, baseline implementation
The scalar implementation is fairly straightforward and merely unrolled
four times.  The baseline implementation closely follows D41971 with
appropriate extensions and extra code paths to pay attention to string
length.

Performance is quite good.  We beat both glibc (except for very long
strings, but they likely use AVX which we don't) and Bionic (except for
medium-sized aligned strings, where we are still in the same ballpark).

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D42122
2023-12-25 14:55:13 +01:00
Robert Clausecker
459ddefcc9 lib/libc/tests/string: add unit tests for strncmp(3)
These are patterned after the previously added (D41970)
strcmp tests, but are extended to check for various length
conditions.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D42122
2023-12-25 14:55:08 +01:00
Robert Clausecker
8b60e1fdbe share/man/man7/simd.7: document amd64 SIMD use for strpbrk()
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision: https://reviews.freebsd.org/D41980
2023-12-25 14:55:04 +01:00
Robert Clausecker
f4fc317c36 lib/libc/amd64/string: implement strpbrk() through strcspn()
This lets us use our optimised strcspn() routine for strpbrk() calls.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D41980
2023-12-25 14:54:58 +01:00
Robert Clausecker
47adb1e012 share/man/man7/simd.7: document new amd64 baseline strcmp()
Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D41971
2023-12-25 14:54:50 +01:00
Robert Clausecker
bca25680b9 lib/libc/amd64/string/strcmp.S: add baseline implementation
This is the most complicated one so far.  The basic idea is to process
the bulk of the string in aligned blocks of 16 bytes such that one
string runs ahead and the other runs behind.  The string that runs ahead
is checked for NUL bytes, the one that runs behind is compared with the
corresponding chunk of the string that runs ahead.  This trades an extra
load per iteration for the very complicated block-reassembly needed in
the other implementations (bionic, glibc).  On the flip side, we need
two code paths depending on the relative alignment of the two buffers.

The initial part of the string is compared directly if it is known not
to cross a page boundary.  Otherwise, a complex slow path to avoid
crossing into unmapped memory commences.

Performance-wise we beat bionic for misaligned strings (i.e. the strings
do not share an alignment offset) and reach comparable performance for
aligned strings.  glibc is a bit better as it has a special kernel for
AVX-512, where this stuff is a bit easier to do.

Sponsored by:	The FreeBSD Foundation
Tested by:	developers@, exp-run
Approved by:	mjg
MFC after:	1 month
MFC to:		stable/14
PR:		275785
Differential Revision:	https://reviews.freebsd.org/D41971
2023-12-25 14:54:33 +01:00
Alexander Motin
13037eaabe ig4: Actively use FIFO thresholds
Before every wait for FIFO interrupt set how much data/space do we
want to see there.  Previous code was not using it for receive, as
result aggregating interrupts only within processing latency.  The
new code needs only one interrupt per transfer per FIFO length.

On my Dell XPS 13 9310 with iichid(4) touchscreen and touchpad this
reduces the interrupt rate per device down to 2 per sample or 16-20
per second when idle and 120-160 per second when actively touched.

MFC after:	1 month
2023-12-24 18:18:11 -05:00
Vladimir Kondratyev
dbca442414 LinuxKPI: Constify src parameter of bitmap_copy
in bitmap_from_arr32() to fix build on 32 bit archs.

Sponsored by:	Serenity Cyber Security, LLC
Fixes:	5ae2e6f913 ("LinuxKPI: Add bitmap_intersects(), bitmap_from_arr32()")
MFC after:	1 week
2023-12-24 15:48:06 +03:00
Vladimir Kondratyev
73ccd188c4 LinuxKPI: Define ioread64() in linux/io.h only on 64 bit archs.
32 bit archs includes nonatomic version from linux/io-64-nonatomic-*.h

Sponsored by:   Serenity Cyber Security, LLC
Fixes:		dcfc983373 ("LinuxKPI: Implement ioread64()")
MFC after:      1 week
2023-12-24 15:48:06 +03:00
Vladimir Kondratyev
b4efc62776 LinuxKPI: Do not use explicit context in FPU sections on powerpc64
It is not supported yet.

Sponsored by:	Serenity Cyber Security, LLC
Fixes:	5a3bd28167 ("LinuxKPI: Add explicit software context to FPU sections")
MFC after:	1 week
2023-12-24 15:48:06 +03:00
Vladimir Kondratyev
9bf957fc9b Bump __FreeBSD_version after LinuxKPI changes.
Sponsored by:	Serenity Cyber Security, LLC
2023-12-24 11:20:01 +03:00
Vladimir Kondratyev
5a3bd28167 LinuxKPI: Add explicit software context to FPU sections
Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This does not work
currently on FreeBSD as its malloc function can not be run within a
critical section. Allocate memory for FPU context to overcome such
restriction.

Sponsored by:	Serenity Cyber Security, LLC
Reviewed by:	manu (previous version), markj
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42822
2023-12-24 11:20:01 +03:00
Vladimir Kondratyev
04952a9456 LinuxKPI: Add acpi_dev_present() function.
acpi_dev_present detects that a given ACPI device is present based on
Hardware ID, Unique ID and Hardware Revision of the device.

Sponsored by:	Serenity Cyber Security, LLC
Reviewed by:	manu
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42823
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
808ae4e29b LinuxKPI: Add pcie_capability_clear_and_set_word() function
It does a Read-Modify-Write operation using clear and set bitmasks on
PCI Express Capability Register at pos. As certain PCI Express
Capability Registers are accessed concurrently in RMW fashion, hence
require locking which is handled transparently to the caller.

Sponsored by:	Serenity CyberSecurity, LLC
Reviewed by:	manu, bz
MFC after:	1 week
MFC TODO:	Move pcie_cap_lock to bottom to preserve KBI compatibility
Differential Revision:	https://reviews.freebsd.org/D42821
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
b8c88a6175 LinuxKPI: Add x86_vendor field to struct cpuinfo_x86
and initialize it at linuxkpi module load.

Sponsored by:	Serenity Cyber Security, LLC
Reviewed by:	manu
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42820
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
730387b64e LinuxKPI: Add linux/apple-gmux.h
Sponsored by:	Serenity Cyber Security, LLC
Obtained from:	OpenBSD
Reviewed by:	manu, bz
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42819
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
c58ddc2546 LinuxKPI: Rename linux_list_add to __list_add() in linux/list.h
to match Linux

Sponsored by:	Serenity Cyber Security, LLC
Reviewed by:	manu, bz
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42818
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
9903d83d67 LinuxKPI: Add linux/ioport.h header
Sponsored by:	Serenity Cyber Security, LLC
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42817
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
7f88d742e5 LinuxKPI: Add get_random_u64() function
Sponsored by:	Serenity Cyber Security, LLC
Reviewed by:	manu, bz
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42816
2023-12-24 11:20:00 +03:00
Vladimir Kondratyev
2b037743a7 LinuxKPI: Add linux/dynamic_debug.h header
Sponsored by:	Serenity Cyber Security, LLC
Obtained from:	OpenBSD
Reviewed by:	manu
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D42815
2023-12-24 11:20:00 +03:00