mirror/freebsd
1
0
Fork 0
mirror of https://git.FreeBSD.org/src.git synced 2024-10-19 02:29:40 +00:00
Code Issues Releases Activity
62ace21a14
freebsd/include/sys
History
Alexander Motin 8469b5aac0
Another set of vdev queue optimizations. 
Switch FIFO queues (SYNC/TRIM) and active queue of vdev queue from
time-sorted AVL-trees to simple lists.  AVL-trees are too expensive
for such a simple task.  To change I/O priority without searching
through the trees, add io_queue_state field to struct zio.

To not check number of queued I/Os for each priority add vq_cqueued
bitmap to struct vdev_queue.  Update it when adding/removing I/Os.
Make vq_cactive a separate array instead of struct vdev_queue_class
member.  Together those allow to avoid lots of cache misses when
looking for work in vdev_queue_class_to_issue().

Introduce deadline of ~0.5s for LBA-sorted queues.  Before this I
saw some I/Os waiting in a queue for up to 8 seconds and possibly
more due to starvation.  With this change I no longer see it.  I
had to slightly more complicate the comparison function, but since
it uses all the same cache lines the difference is minimal.  For a
sequential I/Os the new code in vdev_queue_io_to_issue() actually
often uses more simple avl_first(), falling back to avl_find() and
avl_nearest() only when needed.

Arrange members in struct zio to access only one cache line when
searching through vdev queues.  While there, remove io_alloc_node,
reusing the io_queue_node instead.  Those two are never used same
time.

Remove zfs_vdev_aggregate_trim parameter.  It was disabled for 4
years since implemented, while still wasted time maintaining the
offset-sorted tree of TRIM requests.  Just remove the tree.

Remove locking from txg_all_lists_empty().  It is racy by design,
while 2 pair of locks/unlocks take noticeable time under the vdev
queue lock.

With these changes in my tests with volblocksize=4KB I measure vdev
queue lock spin time reduction by 50% on read and 75% on write.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes #14925
2023-06-27 09:09:48 -07:00
..
crypto Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
fm Configure zed's diagnosis engine with vdev properties 2023-01-23 13:14:25 -08:00
fs Teach zpool scrub to scrub only blocks in error log 2023-05-18 11:59:42 -07:00
lua autoconf: single-step includes 2022-05-10 10:18:51 -07:00
sysevent Teach zpool scrub to scrub only blocks in error log 2023-05-18 11:59:42 -07:00
zstd Unbreak zstd build on sparc64 2022-05-25 09:18:49 -07:00
abd_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
abd.h Use __attribute__((malloc)) on memory allocation functions 2023-05-26 15:47:52 -07:00
aggsum.h More aggsum optimizations 2021-06-07 09:02:47 -07:00
arc_impl.h Remove ARC/ZIO physdone callbacks. 2023-06-15 10:49:03 -07:00
arc.h Remove ARC/ZIO physdone callbacks. 2023-06-15 10:49:03 -07:00
asm_linkage.h Unify Assembler files between Linux and Windows 2023-01-17 11:09:19 -08:00
avl_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
avl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
bitmap.h Replace dead opensolaris.org license links 2023-03-14 14:44:01 -07:00
bitops.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
blake3.h Update BLAKE3 for using the new impl handling 2023-03-02 13:52:27 -08:00
blkptr.h OpenZFS 8067 - zdb should be able to dump literal embedded block pointer 2017-07-07 11:28:01 -07:00
bplist.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
bpobj.h Prefetch on deadlists merge 2023-01-25 11:30:24 -08:00
bptree.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
bqueue.h Batch enqueue/dequeue for bqueue 2023-01-10 13:39:22 -08:00
brt.h Replace dead opensolaris.org license links 2023-03-14 14:44:01 -07:00
btree.h btree: Implement faster binary search algorithm 2023-05-26 10:03:12 -07:00
dataset_kstats.h Add support for per dataset zil stats and use wmsum counters 2022-07-20 17:14:06 -07:00
dbuf.h Additional block cloning fixes. 2023-05-11 16:06:36 -07:00
ddt.h Implementation of block cloning for ZFS 2023-03-10 11:59:53 -08:00
dmu_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dmu_objset.h Silence clang warning of flexible array not at end 2023-04-18 18:10:40 -07:00
dmu_recv.h nvpair: Constify string functions 2023-03-14 15:25:50 -07:00
dmu_redact.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dmu_send.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dmu_traverse.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dmu_tx.h Add dmu_tx_hold_append() interface 2023-05-09 09:03:10 -07:00
dmu_zfetch.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
dmu.h Teach zpool scrub to scrub only blocks in error log 2023-05-18 11:59:42 -07:00
dnode.h Linux 6.3 compat: Fix memcpy "detected field-spanning write" error 2023-04-13 09:12:03 -07:00
dsl_bookmark.h Remaining {=> const} char|void *tag 2022-06-29 14:08:59 -07:00
dsl_crypt.h Fix zpool status in case of unloaded keys 2022-08-22 17:42:01 -07:00
dsl_dataset.h Revert zfeature_active() to static 2023-02-28 14:03:52 -08:00
dsl_deadlist.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
dsl_deleg.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_destroy.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_dir.h Cleanup ->dd_space_towrite should be unsigned 2023-01-20 11:10:15 -08:00
dsl_pool.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
dsl_prop.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_scan.h Teach zpool scrub to scrub only blocks in error log 2023-05-18 11:59:42 -07:00
dsl_synctask.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
dsl_userhold.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
edonr.h Remove unused constant EdonR256_BLOCK_BITSIZE 2023-03-22 08:39:48 -07:00
efi_partition.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
frame.h Linux 5.10 compat: frame.h renamed objtool.h 2020-11-02 22:01:10 +00:00
hkdf.h Encryption patch follow-up 2017-10-11 16:54:48 -04:00
metaslab_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
metaslab.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
mmp.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
mntent.h Expose ZFS dataset case sensitivity setting via sb_opts 2022-07-14 10:38:16 -07:00
mod.h linux: module: weld all but spl.ko into zfs.ko 2022-04-20 13:28:24 -07:00
multilist.h Re-embed multilist_t storage 2021-06-10 10:42:31 -06:00
nvpair_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
nvpair.h nvpair: Use flexible array member for nvpair name strings 2023-03-14 15:25:55 -07:00
objlist.h Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
pathname.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
qat.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
range_tree.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
rrwlock.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
sa_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
sa.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
sha2.h Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
skein.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
spa_checkpoint.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
spa_checksum.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
spa_impl.h Teach zpool scrub to scrub only blocks in error log 2023-05-18 11:59:42 -07:00
spa_log_spacemap.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
spa.h Finally drop long disabled vdev cache. 2023-06-09 12:40:55 -07:00
space_map.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
space_reftree.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
sysevent.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
txg_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
txg.h Cleanup: Specify unsignedness on things that should not be signed 2022-09-27 16:42:41 -07:00
u8_textprep_data.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
u8_textprep.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
uberblock_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
uberblock.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
uio_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
unique.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
uuid.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_disk.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_draid.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_file.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_impl.h Another set of vdev queue optimizations. 2023-06-27 09:09:48 -07:00
vdev_indirect_births.h OpenZFS 7614, 9064 - zfs device evacuation/removal 2018-04-14 12:16:17 -07:00
vdev_indirect_mapping.h OpenZFS 7614, 9064 - zfs device evacuation/removal 2018-04-14 12:16:17 -07:00
vdev_initialize.h Add the ability to uninitialize 2023-05-18 10:02:20 -07:00
vdev_raidz_impl.h Cleanup Raid-Z Typo fixes 2022-09-06 09:43:21 -07:00
vdev_raidz.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_rebuild.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
vdev_removal.h Cleanup: Specify unsignedness on things that should not be signed 2022-09-27 16:42:41 -07:00
vdev_trim.h Fix short-lived txg caused by autotrim 2023-03-28 08:43:41 -07:00
vdev.h Another set of vdev queue optimizations. 2023-06-27 09:09:48 -07:00
xvattr.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zap_impl.h Optimize microzaps 2022-10-20 11:57:15 -07:00
zap_leaf.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zap.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zcp_global.h OpenZFS 7431 - ZFS Channel Programs 2018-02-08 15:28:18 -08:00
zcp_iter.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zcp_prop.h OpenZFS 7431 - ZFS Channel Programs 2018-02-08 15:28:18 -08:00
zcp_set.h Support setting user properties in a channel program 2020-02-14 13:41:42 -08:00
zcp.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
zfeature.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_acl.h Linux 6.3 compat: idmapped mount API changes 2023-04-10 14:15:36 -07:00
zfs_bootenv.h zfs label bootenv should store data as nvlist 2020-09-15 15:42:27 -07:00
zfs_chksum.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_context.h Introduce kmem_scnprintf() 2022-10-29 13:05:11 -07:00
zfs_debug.h Implementation of block cloning for ZFS 2023-03-10 11:59:53 -08:00
zfs_delay.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_file.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_fuid.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_impl.h Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
zfs_ioctl_impl.h Cleanup: 64-bit kernel module parameters should use fixed width types 2022-10-13 10:03:29 -07:00
zfs_ioctl.h DMU_BACKUP_FEATURE: indicate that bit 28 and 29 are reserved 2022-09-27 16:55:32 -07:00
zfs_onexit.h zfs_onexit_add_cb: make action_handle point to a uintptr_t 2022-11-03 09:52:12 -07:00
zfs_project.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_quota.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_racct.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_ratelimit.h Change checksum & IO delay ratelimit values 2018-03-04 17:34:51 -08:00
zfs_refcount.h Switch refcount tracking from lists to AVL-trees. 2023-06-14 08:02:27 -07:00
zfs_rlock.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_sa.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_stat.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_sysfs.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_vfsops.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zfs_vnops.h Implementation of block cloning for ZFS 2023-03-10 11:59:53 -08:00
zfs_znode.h znode: expose zfs_get_zplprop to libzpool 2023-06-05 11:53:44 -07:00
zil_impl.h ZIL: Reduce scope of per-dataset zl_issuer_lock. 2023-05-25 09:48:43 -07:00
zil.h zil: Add some more statistics. 2023-05-25 13:51:53 -07:00
zio_checksum.h Add generic implementation handling and SHA2 impl 2023-03-02 13:52:21 -08:00
zio_compress.h Skip memory allocation when compressing holes 2023-02-27 14:41:02 -08:00
zio_crypt.h Enable -Wwrite-strings 2022-06-29 14:08:54 -07:00
zio_impl.h Implementation of block cloning for ZFS 2023-03-10 11:59:53 -08:00
zio_priority.h Add device rebuild feature 2020-07-03 11:05:50 -07:00
zio.h Another set of vdev queue optimizations. 2023-06-27 09:09:48 -07:00
zrlock.h Pack zrlock_t by 8 bytes 2023-01-05 09:31:55 -08:00
zthr.h Avoid memory allocations in the ARC eviction thread 2022-01-21 10:28:13 -08:00
zvol_impl.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00
zvol.h Replace dead opensolaris.org license link 2022-07-11 14:16:13 -07:00