MFuser/delphij/zfs-arc-rebase@r281754:

In r256613, taskqueue_enqueue_locked() have been modified to release the
task queue lock before returning.  In r276665, taskqueue_drain_all() will
call taskqueue_enqueue_locked() to insert the barrier task into the queue,
but did not reacquire the lock after it but later code expects the lock
still being held (e.g. TQ_SLEEP()).

The barrier task is special and if we release then reacquire the lock,
there would be a small race window where a high priority task could sneak
into the queue.  Looking more closely, the race seems to be tolerable but
is undesirable from semantics standpoint.

To solve this, in taskqueue_drain_tq_queue(), instead of directly calling
taskqueue_enqueue_locked(), insert the barrier task directly without
releasing the lock.

sys/kern/subr_taskqueue.c

@@ -323,17 +323,16 @@ taskqueue_drain_tq_queue(struct taskqueue *queue)
 		return;
 
 	/*
-	 * Enqueue our barrier with the lowest possible priority
+	 * Enqueue our barrier after all current tasks, but with
-	 * so we are inserted after all current tasks.
+	 * the highest priority so that newly queued tasks cannot
+	 * pass it.  Because of the high priority, we can not use
+	 * taskqueue_enqueue_locked directly (which drops the lock
+	 * anyway) so just insert it at tail while we have the
+	 * queue lock.
 	 */
-	TASK_INIT(&t_barrier, 0, taskqueue_task_nop_fn, &t_barrier);
+	TASK_INIT(&t_barrier, USHRT_MAX, taskqueue_task_nop_fn, &t_barrier);
-	taskqueue_enqueue_locked(queue, &t_barrier);
+	STAILQ_INSERT_TAIL(&queue->tq_queue, &t_barrier, ta_link);
-
+	t_barrier.ta_pending = 1;
-	/*
- 	 * Raise the barrier's priority so newly queued tasks cannot
- 	 * pass it.
- 	 */
-	t_barrier.ta_priority = USHRT_MAX;
 
 	/*
 	 * Once the barrier has executed, all previously queued tasks