2003-01-26 05:23:15 +00:00
|
|
|
/*-
|
2003-04-11 03:47:14 +00:00
|
|
|
* Copyright (c) 2002-2003, Jeffrey Roberson <jeff@freebsd.org>
|
2003-01-26 05:23:15 +00:00
|
|
|
* All rights reserved.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice unmodified, this list of conditions, and the following
|
|
|
|
* disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
|
|
|
|
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
|
|
|
|
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
|
|
* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
|
|
|
|
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
|
|
|
|
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
|
|
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
|
|
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
|
|
|
|
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*/
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
2004-09-02 18:59:15 +00:00
|
|
|
#include <opt_sched.h>
|
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
#define kse td_sched
|
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
#include <sys/param.h>
|
|
|
|
#include <sys/systm.h>
|
2004-07-10 21:38:22 +00:00
|
|
|
#include <sys/kdb.h>
|
2003-01-26 05:23:15 +00:00
|
|
|
#include <sys/kernel.h>
|
|
|
|
#include <sys/ktr.h>
|
|
|
|
#include <sys/lock.h>
|
|
|
|
#include <sys/mutex.h>
|
|
|
|
#include <sys/proc.h>
|
2003-04-02 06:46:43 +00:00
|
|
|
#include <sys/resource.h>
|
2003-11-04 07:45:41 +00:00
|
|
|
#include <sys/resourcevar.h>
|
2003-01-26 05:23:15 +00:00
|
|
|
#include <sys/sched.h>
|
|
|
|
#include <sys/smp.h>
|
|
|
|
#include <sys/sx.h>
|
|
|
|
#include <sys/sysctl.h>
|
|
|
|
#include <sys/sysproto.h>
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
#include <sys/turnstile.h>
|
2003-01-26 05:23:15 +00:00
|
|
|
#include <sys/vmmeter.h>
|
|
|
|
#ifdef KTRACE
|
|
|
|
#include <sys/uio.h>
|
|
|
|
#include <sys/ktrace.h>
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#include <machine/cpu.h>
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#include <machine/smp.h>
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
/* decay 95% of `p_pctcpu' in 60 seconds; see CCPU_SHIFT before changing */
|
|
|
|
/* XXX This is bogus compatability crap for ps */
|
|
|
|
static fixpt_t ccpu = 0.95122942450071400909 * FSCALE; /* exp(-1/20) */
|
|
|
|
SYSCTL_INT(_kern, OID_AUTO, ccpu, CTLFLAG_RD, &ccpu, 0, "");
|
|
|
|
|
|
|
|
static void sched_setup(void *dummy);
|
|
|
|
SYSINIT(sched_setup, SI_SUB_RUN_QUEUE, SI_ORDER_FIRST, sched_setup, NULL)
|
|
|
|
|
2004-07-23 23:09:00 +00:00
|
|
|
static SYSCTL_NODE(_kern, OID_AUTO, sched, CTLFLAG_RW, 0, "Scheduler");
|
2003-04-11 03:47:14 +00:00
|
|
|
|
2004-07-23 23:09:00 +00:00
|
|
|
SYSCTL_STRING(_kern_sched, OID_AUTO, name, CTLFLAG_RD, "ule", 0,
|
|
|
|
"Scheduler name");
|
2004-06-21 22:05:46 +00:00
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
static int slice_min = 1;
|
|
|
|
SYSCTL_INT(_kern_sched, OID_AUTO, slice_min, CTLFLAG_RW, &slice_min, 0, "");
|
|
|
|
|
2003-06-15 02:18:29 +00:00
|
|
|
static int slice_max = 10;
|
2003-04-11 03:47:14 +00:00
|
|
|
SYSCTL_INT(_kern_sched, OID_AUTO, slice_max, CTLFLAG_RW, &slice_max, 0, "");
|
|
|
|
|
|
|
|
int realstathz;
|
|
|
|
int tickincr = 1;
|
2003-02-10 14:11:23 +00:00
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
/*
|
|
|
|
* The schedulable entity that can be given a context to run.
|
|
|
|
* A process may have several of these. Probably one per processor
|
|
|
|
* but posibly a few more. In this universe they are grouped
|
|
|
|
* with a KSEG that contains the priority and niceness
|
|
|
|
* for the group.
|
|
|
|
*/
|
|
|
|
struct kse {
|
|
|
|
TAILQ_ENTRY(kse) ke_procq; /* (j/z) Run queue. */
|
|
|
|
int ke_flags; /* (j) KEF_* flags. */
|
|
|
|
struct thread *ke_thread; /* (*) Active associated thread. */
|
|
|
|
fixpt_t ke_pctcpu; /* (j) %cpu during p_swtime. */
|
|
|
|
char ke_rqindex; /* (j) Run queue index. */
|
|
|
|
enum {
|
|
|
|
KES_THREAD = 0x0, /* slaved to thread state */
|
|
|
|
KES_ONRUNQ
|
|
|
|
} ke_state; /* (j) thread sched specific status. */
|
|
|
|
int ke_slptime;
|
|
|
|
int ke_slice;
|
|
|
|
struct runq *ke_runq;
|
|
|
|
u_char ke_cpu; /* CPU that we have affinity for. */
|
|
|
|
/* The following variables are only used for pctcpu calculation */
|
|
|
|
int ke_ltick; /* Last tick that we were running on */
|
|
|
|
int ke_ftick; /* First tick that we were running on */
|
|
|
|
int ke_ticks; /* Tick count */
|
|
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
#define td_kse td_sched
|
|
|
|
#define td_slptime td_kse->ke_slptime
|
|
|
|
#define ke_proc ke_thread->td_proc
|
|
|
|
#define ke_ksegrp ke_thread->td_ksegrp
|
|
|
|
|
|
|
|
/* flags kept in ke_flags */
|
|
|
|
#define KEF_SCHED0 0x00001 /* For scheduler-specific use. */
|
|
|
|
#define KEF_SCHED1 0x00002 /* For scheduler-specific use. */
|
|
|
|
#define KEF_SCHED2 0x00004 /* For scheduler-specific use. */
|
|
|
|
#define KEF_SCHED3 0x00008 /* For scheduler-specific use. */
|
2004-12-14 10:34:27 +00:00
|
|
|
#define KEF_SCHED4 0x00010
|
|
|
|
#define KEF_SCHED5 0x00020
|
2004-09-05 02:09:54 +00:00
|
|
|
#define KEF_DIDRUN 0x02000 /* Thread actually ran. */
|
|
|
|
#define KEF_EXIT 0x04000 /* Thread is being killed. */
|
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
|
|
|
* These datastructures are allocated within their parent datastructure but
|
|
|
|
* are scheduler specific.
|
|
|
|
*/
|
|
|
|
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#define ke_assign ke_procq.tqe_next
|
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
#define KEF_ASSIGNED 0x0001 /* Thread is being migrated. */
|
|
|
|
#define KEF_BOUND 0x0002 /* Thread can not migrate. */
|
|
|
|
#define KEF_XFERABLE 0x0004 /* Thread was added as transferable. */
|
|
|
|
#define KEF_HOLD 0x0008 /* Thread is temporarily bound. */
|
|
|
|
#define KEF_REMOVED 0x0010 /* Thread was removed while ASSIGNED */
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
#define KEF_INTERNAL 0x0020
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
struct kg_sched {
|
2004-09-05 02:09:54 +00:00
|
|
|
struct thread *skg_last_assigned; /* (j) Last thread assigned to */
|
|
|
|
/* the system scheduler */
|
2003-02-10 14:03:45 +00:00
|
|
|
int skg_slptime; /* Number of ticks we vol. slept */
|
|
|
|
int skg_runtime; /* Number of ticks we were running */
|
2004-09-05 02:09:54 +00:00
|
|
|
int skg_avail_opennings; /* (j) Num unfilled slots in group.*/
|
|
|
|
int skg_concurrency; /* (j) Num threads requested in group.*/
|
2003-01-26 05:23:15 +00:00
|
|
|
};
|
2004-09-05 02:09:54 +00:00
|
|
|
#define kg_last_assigned kg_sched->skg_last_assigned
|
|
|
|
#define kg_avail_opennings kg_sched->skg_avail_opennings
|
|
|
|
#define kg_concurrency kg_sched->skg_concurrency
|
|
|
|
#define kg_runtime kg_sched->skg_runtime
|
|
|
|
#define kg_slptime kg_sched->skg_slptime
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-10-05 21:10:44 +00:00
|
|
|
#define SLOT_RELEASE(kg) \
|
|
|
|
do { \
|
|
|
|
kg->kg_avail_opennings++; \
|
|
|
|
CTR3(KTR_RUNQ, "kg %p(%d) Slot released (->%d)", \
|
|
|
|
kg, \
|
|
|
|
kg->kg_concurrency, \
|
|
|
|
kg->kg_avail_opennings); \
|
|
|
|
/*KASSERT((kg->kg_avail_opennings <= kg->kg_concurrency), \
|
|
|
|
("slots out of whack")); */ \
|
|
|
|
} while (0)
|
|
|
|
|
|
|
|
#define SLOT_USE(kg) \
|
|
|
|
do { \
|
|
|
|
kg->kg_avail_opennings--; \
|
|
|
|
CTR3(KTR_RUNQ, "kg %p(%d) Slot used (->%d)", \
|
|
|
|
kg, \
|
|
|
|
kg->kg_concurrency, \
|
|
|
|
kg->kg_avail_opennings); \
|
|
|
|
/*KASSERT((kg->kg_avail_opennings >= 0), \
|
|
|
|
("slots out of whack"));*/ \
|
|
|
|
} while (0)
|
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
static struct kse kse0;
|
|
|
|
static struct kg_sched kg_sched0;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
/*
|
2003-06-21 02:22:47 +00:00
|
|
|
* The priority is primarily determined by the interactivity score. Thus, we
|
|
|
|
* give lower(better) priorities to kse groups that use less CPU. The nice
|
|
|
|
* value is then directly added to this to allow nice to have some effect
|
|
|
|
* on latency.
|
2003-03-04 02:45:59 +00:00
|
|
|
*
|
|
|
|
* PRI_RANGE: Total priority range for timeshare threads.
|
2003-06-21 02:22:47 +00:00
|
|
|
* PRI_NRESV: Number of nice values.
|
2003-03-04 02:45:59 +00:00
|
|
|
* PRI_BASE: The start of the dynamic range.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-03-04 02:45:59 +00:00
|
|
|
#define SCHED_PRI_RANGE (PRI_MAX_TIMESHARE - PRI_MIN_TIMESHARE + 1)
|
2003-11-02 03:49:32 +00:00
|
|
|
#define SCHED_PRI_NRESV ((PRIO_MAX - PRIO_MIN) + 1)
|
|
|
|
#define SCHED_PRI_NHALF (SCHED_PRI_NRESV / 2)
|
2003-06-21 02:22:47 +00:00
|
|
|
#define SCHED_PRI_BASE (PRI_MIN_TIMESHARE)
|
2003-04-11 03:47:14 +00:00
|
|
|
#define SCHED_PRI_INTERACT(score) \
|
2003-06-21 02:22:47 +00:00
|
|
|
((score) * SCHED_PRI_RANGE / SCHED_INTERACT_MAX)
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
/*
|
2003-03-04 02:45:59 +00:00
|
|
|
* These determine the interactivity of a process.
|
2003-01-26 05:23:15 +00:00
|
|
|
*
|
2003-02-10 14:03:45 +00:00
|
|
|
* SLP_RUN_MAX: Maximum amount of sleep time + run time we'll accumulate
|
|
|
|
* before throttling back.
|
2003-11-02 03:36:33 +00:00
|
|
|
* SLP_RUN_FORK: Maximum slp+run time to inherit at fork time.
|
2003-06-15 02:18:29 +00:00
|
|
|
* INTERACT_MAX: Maximum interactivity value. Smaller is better.
|
2003-03-04 02:45:59 +00:00
|
|
|
* INTERACT_THRESH: Threshhold for placement on the current runq.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-10-16 08:17:43 +00:00
|
|
|
#define SCHED_SLP_RUN_MAX ((hz * 5) << 10)
|
2003-11-02 03:36:33 +00:00
|
|
|
#define SCHED_SLP_RUN_FORK ((hz / 2) << 10)
|
2003-06-15 02:18:29 +00:00
|
|
|
#define SCHED_INTERACT_MAX (100)
|
|
|
|
#define SCHED_INTERACT_HALF (SCHED_INTERACT_MAX / 2)
|
2003-10-16 08:17:43 +00:00
|
|
|
#define SCHED_INTERACT_THRESH (30)
|
2003-03-04 02:45:59 +00:00
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
|
|
|
* These parameters and macros determine the size of the time slice that is
|
|
|
|
* granted to each thread.
|
|
|
|
*
|
|
|
|
* SLICE_MIN: Minimum time slice granted, in units of ticks.
|
|
|
|
* SLICE_MAX: Maximum time slice granted.
|
|
|
|
* SLICE_RANGE: Range of available time slices scaled by hz.
|
2003-04-02 06:46:43 +00:00
|
|
|
* SLICE_SCALE: The number slices granted per val in the range of [0, max].
|
|
|
|
* SLICE_NICE: Determine the amount of slice granted to a scaled nice.
|
2003-11-02 04:10:15 +00:00
|
|
|
* SLICE_NTHRESH: The nice cutoff point for slice assignment.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-04-11 03:47:14 +00:00
|
|
|
#define SCHED_SLICE_MIN (slice_min)
|
|
|
|
#define SCHED_SLICE_MAX (slice_max)
|
2004-02-01 10:38:13 +00:00
|
|
|
#define SCHED_SLICE_INTERACTIVE (slice_max)
|
2003-11-02 04:10:15 +00:00
|
|
|
#define SCHED_SLICE_NTHRESH (SCHED_PRI_NHALF - 1)
|
2003-03-04 02:45:59 +00:00
|
|
|
#define SCHED_SLICE_RANGE (SCHED_SLICE_MAX - SCHED_SLICE_MIN + 1)
|
2003-01-26 05:23:15 +00:00
|
|
|
#define SCHED_SLICE_SCALE(val, max) (((val) * SCHED_SLICE_RANGE) / (max))
|
2003-04-02 06:46:43 +00:00
|
|
|
#define SCHED_SLICE_NICE(nice) \
|
2003-11-02 04:10:15 +00:00
|
|
|
(SCHED_SLICE_MAX - SCHED_SLICE_SCALE((nice), SCHED_SLICE_NTHRESH))
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
/*
|
2004-09-05 02:09:54 +00:00
|
|
|
* This macro determines whether or not the thread belongs on the current or
|
2003-01-26 05:23:15 +00:00
|
|
|
* next run queue.
|
|
|
|
*/
|
2003-04-11 03:47:14 +00:00
|
|
|
#define SCHED_INTERACTIVE(kg) \
|
|
|
|
(sched_interact_score(kg) < SCHED_INTERACT_THRESH)
|
2003-04-12 22:33:24 +00:00
|
|
|
#define SCHED_CURR(kg, ke) \
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
((ke->ke_thread->td_flags & TDF_BORROWING) || SCHED_INTERACTIVE(kg))
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Cpu percentage computation macros and defines.
|
|
|
|
*
|
|
|
|
* SCHED_CPU_TIME: Number of seconds to average the cpu usage across.
|
|
|
|
* SCHED_CPU_TICKS: Number of hz ticks to average the cpu usage across.
|
|
|
|
*/
|
|
|
|
|
2003-04-02 08:22:33 +00:00
|
|
|
#define SCHED_CPU_TIME 10
|
2003-01-26 05:23:15 +00:00
|
|
|
#define SCHED_CPU_TICKS (hz * SCHED_CPU_TIME)
|
|
|
|
|
|
|
|
/*
|
2003-04-11 03:47:14 +00:00
|
|
|
* kseq - per processor runqs and statistics.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
|
|
|
struct kseq {
|
2003-04-11 03:47:14 +00:00
|
|
|
struct runq ksq_idle; /* Queue of IDLE threads. */
|
|
|
|
struct runq ksq_timeshare[2]; /* Run queues for !IDLE. */
|
|
|
|
struct runq *ksq_next; /* Next timeshare queue. */
|
|
|
|
struct runq *ksq_curr; /* Current queue. */
|
2003-11-02 10:56:48 +00:00
|
|
|
int ksq_load_timeshare; /* Load for timeshare. */
|
2003-04-11 03:47:14 +00:00
|
|
|
int ksq_load; /* Aggregate load. */
|
2003-11-02 03:49:32 +00:00
|
|
|
short ksq_nice[SCHED_PRI_NRESV]; /* KSEs in each nice bin. */
|
2003-04-11 03:47:14 +00:00
|
|
|
short ksq_nicemin; /* Least nice. */
|
2003-02-03 05:30:07 +00:00
|
|
|
#ifdef SMP
|
2003-12-11 03:57:10 +00:00
|
|
|
int ksq_transferable;
|
|
|
|
LIST_ENTRY(kseq) ksq_siblings; /* Next in kseq group. */
|
|
|
|
struct kseq_group *ksq_group; /* Our processor group. */
|
|
|
|
volatile struct kse *ksq_assigned; /* assigned by another CPU. */
|
2004-02-01 02:48:36 +00:00
|
|
|
#else
|
|
|
|
int ksq_sysload; /* For loadavg, !ITHD load. */
|
2003-02-03 05:30:07 +00:00
|
|
|
#endif
|
2003-01-26 05:23:15 +00:00
|
|
|
};
|
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
#ifdef SMP
|
|
|
|
/*
|
|
|
|
* kseq groups are groups of processors which can cheaply share threads. When
|
|
|
|
* one processor in the group goes idle it will check the runqs of the other
|
|
|
|
* processors in its group prior to halting and waiting for an interrupt.
|
|
|
|
* These groups are suitable for SMT (Symetric Multi-Threading) and not NUMA.
|
|
|
|
* In a numa environment we'd want an idle bitmap per group and a two tiered
|
|
|
|
* load balancer.
|
|
|
|
*/
|
|
|
|
struct kseq_group {
|
|
|
|
int ksg_cpus; /* Count of CPUs in this kseq group. */
|
2004-03-27 18:21:24 +00:00
|
|
|
cpumask_t ksg_cpumask; /* Mask of cpus in this group. */
|
|
|
|
cpumask_t ksg_idlemask; /* Idle cpus in this group. */
|
|
|
|
cpumask_t ksg_mask; /* Bit mask for first cpu. */
|
2003-12-12 07:33:51 +00:00
|
|
|
int ksg_load; /* Total load of this group. */
|
2003-12-11 03:57:10 +00:00
|
|
|
int ksg_transferable; /* Transferable load of this group. */
|
|
|
|
LIST_HEAD(, kseq) ksg_members; /* Linked list of all members. */
|
|
|
|
};
|
|
|
|
#endif
|
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
|
|
|
* One kse queue per processor.
|
|
|
|
*/
|
2003-01-29 07:00:51 +00:00
|
|
|
#ifdef SMP
|
2004-03-27 18:21:24 +00:00
|
|
|
static cpumask_t kseq_idle;
|
2003-12-12 07:33:51 +00:00
|
|
|
static int ksg_maxid;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static struct kseq kseq_cpu[MAXCPU];
|
2003-12-11 03:57:10 +00:00
|
|
|
static struct kseq_group kseq_groups[MAXCPU];
|
2004-06-02 05:46:48 +00:00
|
|
|
static int bal_tick;
|
|
|
|
static int gbal_tick;
|
2004-12-26 22:56:08 +00:00
|
|
|
static int balance_groups;
|
2004-06-02 05:46:48 +00:00
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
#define KSEQ_SELF() (&kseq_cpu[PCPU_GET(cpuid)])
|
|
|
|
#define KSEQ_CPU(x) (&kseq_cpu[(x)])
|
2003-12-12 07:33:51 +00:00
|
|
|
#define KSEQ_ID(x) ((x) - kseq_cpu)
|
|
|
|
#define KSEQ_GROUP(x) (&kseq_groups[(x)])
|
2003-12-11 03:57:10 +00:00
|
|
|
#else /* !SMP */
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static struct kseq kseq_cpu;
|
2004-06-02 05:46:48 +00:00
|
|
|
|
2003-01-29 07:00:51 +00:00
|
|
|
#define KSEQ_SELF() (&kseq_cpu)
|
|
|
|
#define KSEQ_CPU(x) (&kseq_cpu)
|
|
|
|
#endif
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
static void slot_fill(struct ksegrp *kg);
|
|
|
|
static struct kse *sched_choose(void); /* XXX Should be thread * */
|
2003-04-02 06:46:43 +00:00
|
|
|
static void sched_slice(struct kse *ke);
|
2003-04-11 03:47:14 +00:00
|
|
|
static void sched_priority(struct ksegrp *kg);
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
static void sched_thread_priority(struct thread *td, u_char prio);
|
2003-03-04 02:45:59 +00:00
|
|
|
static int sched_interact_score(struct ksegrp *kg);
|
2003-06-17 06:39:51 +00:00
|
|
|
static void sched_interact_update(struct ksegrp *kg);
|
2003-11-02 03:36:33 +00:00
|
|
|
static void sched_interact_fork(struct ksegrp *kg);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void sched_pctcpu_update(struct kse *ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-02-03 05:30:07 +00:00
|
|
|
/* Operations on per processor queues */
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static struct kse * kseq_choose(struct kseq *kseq);
|
2003-01-29 07:00:51 +00:00
|
|
|
static void kseq_setup(struct kseq *kseq);
|
2003-11-15 07:32:07 +00:00
|
|
|
static void kseq_load_add(struct kseq *kseq, struct kse *ke);
|
|
|
|
static void kseq_load_rem(struct kseq *kseq, struct kse *ke);
|
2004-12-26 22:56:08 +00:00
|
|
|
static __inline void kseq_runq_add(struct kseq *kseq, struct kse *ke, int);
|
2003-11-15 07:32:07 +00:00
|
|
|
static __inline void kseq_runq_rem(struct kseq *kseq, struct kse *ke);
|
2003-04-11 03:47:14 +00:00
|
|
|
static void kseq_nice_add(struct kseq *kseq, int nice);
|
|
|
|
static void kseq_nice_rem(struct kseq *kseq, int nice);
|
2003-04-18 05:24:10 +00:00
|
|
|
void kseq_print(int cpu);
|
2003-02-03 05:30:07 +00:00
|
|
|
#ifdef SMP
|
2003-12-11 03:57:10 +00:00
|
|
|
static int kseq_transfer(struct kseq *ksq, struct kse *ke, int class);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static struct kse *runq_steal(struct runq *rq);
|
2004-06-02 05:46:48 +00:00
|
|
|
static void sched_balance(void);
|
|
|
|
static void sched_balance_groups(void);
|
2003-12-12 07:33:51 +00:00
|
|
|
static void sched_balance_group(struct kseq_group *ksg);
|
|
|
|
static void sched_balance_pair(struct kseq *high, struct kseq *low);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void kseq_move(struct kseq *from, int cpu);
|
2003-12-11 03:57:10 +00:00
|
|
|
static int kseq_idled(struct kseq *kseq);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void kseq_notify(struct kse *ke, int cpu);
|
|
|
|
static void kseq_assign(struct kseq *);
|
2003-12-11 03:57:10 +00:00
|
|
|
static struct kse *kseq_steal(struct kseq *kseq, int stealidle);
|
2004-12-26 22:56:08 +00:00
|
|
|
#define KSE_CAN_MIGRATE(ke) \
|
2004-09-11 10:07:22 +00:00
|
|
|
((ke)->ke_thread->td_pinned == 0 && ((ke)->ke_flags & KEF_BOUND) == 0)
|
2003-02-03 05:30:07 +00:00
|
|
|
#endif
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
void
|
2003-04-18 05:24:10 +00:00
|
|
|
kseq_print(int cpu)
|
2003-02-03 05:30:07 +00:00
|
|
|
{
|
2003-04-18 05:24:10 +00:00
|
|
|
struct kseq *kseq;
|
2003-04-11 03:47:14 +00:00
|
|
|
int i;
|
2003-04-03 00:29:28 +00:00
|
|
|
|
2003-04-18 05:24:10 +00:00
|
|
|
kseq = KSEQ_CPU(cpu);
|
2003-04-03 00:29:28 +00:00
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
printf("kseq:\n");
|
|
|
|
printf("\tload: %d\n", kseq->ksq_load);
|
2003-11-15 07:32:07 +00:00
|
|
|
printf("\tload TIMESHARE: %d\n", kseq->ksq_load_timeshare);
|
2003-11-02 10:56:48 +00:00
|
|
|
#ifdef SMP
|
2003-12-11 03:57:10 +00:00
|
|
|
printf("\tload transferable: %d\n", kseq->ksq_transferable);
|
2003-11-02 10:56:48 +00:00
|
|
|
#endif
|
2003-04-11 03:47:14 +00:00
|
|
|
printf("\tnicemin:\t%d\n", kseq->ksq_nicemin);
|
|
|
|
printf("\tnice counts:\n");
|
2003-11-02 03:49:32 +00:00
|
|
|
for (i = 0; i < SCHED_PRI_NRESV; i++)
|
2003-04-11 03:47:14 +00:00
|
|
|
if (kseq->ksq_nice[i])
|
|
|
|
printf("\t\t%d = %d\n",
|
|
|
|
i - SCHED_PRI_NHALF, kseq->ksq_nice[i]);
|
|
|
|
}
|
2003-04-03 00:29:28 +00:00
|
|
|
|
2003-11-15 07:32:07 +00:00
|
|
|
static __inline void
|
2004-12-26 22:56:08 +00:00
|
|
|
kseq_runq_add(struct kseq *kseq, struct kse *ke, int flags)
|
2003-11-15 07:32:07 +00:00
|
|
|
{
|
|
|
|
#ifdef SMP
|
2004-12-26 22:56:08 +00:00
|
|
|
if (KSE_CAN_MIGRATE(ke)) {
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq->ksq_transferable++;
|
|
|
|
kseq->ksq_group->ksg_transferable++;
|
2004-08-10 07:52:21 +00:00
|
|
|
ke->ke_flags |= KEF_XFERABLE;
|
2003-12-11 03:57:10 +00:00
|
|
|
}
|
2003-11-15 07:32:07 +00:00
|
|
|
#endif
|
2004-12-26 22:56:08 +00:00
|
|
|
runq_add(ke->ke_runq, ke, flags);
|
2003-11-15 07:32:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static __inline void
|
|
|
|
kseq_runq_rem(struct kseq *kseq, struct kse *ke)
|
|
|
|
{
|
|
|
|
#ifdef SMP
|
2004-08-10 07:52:21 +00:00
|
|
|
if (ke->ke_flags & KEF_XFERABLE) {
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq->ksq_transferable--;
|
|
|
|
kseq->ksq_group->ksg_transferable--;
|
2004-08-10 07:52:21 +00:00
|
|
|
ke->ke_flags &= ~KEF_XFERABLE;
|
2003-12-11 03:57:10 +00:00
|
|
|
}
|
2003-11-15 07:32:07 +00:00
|
|
|
#endif
|
|
|
|
runq_remove(ke->ke_runq, ke);
|
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
static void
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_add(struct kseq *kseq, struct kse *ke)
|
2003-04-11 03:47:14 +00:00
|
|
|
{
|
2003-11-02 10:56:48 +00:00
|
|
|
int class;
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-11-02 10:56:48 +00:00
|
|
|
class = PRI_BASE(ke->ke_ksegrp->kg_pri_class);
|
|
|
|
if (class == PRI_TIMESHARE)
|
|
|
|
kseq->ksq_load_timeshare++;
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq->ksq_load++;
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR1(KTR_SCHED, "load: %d", kseq->ksq_load);
|
2004-04-22 21:37:46 +00:00
|
|
|
if (class != PRI_ITHD && (ke->ke_proc->p_flag & P_NOLOAD) == 0)
|
2004-02-01 02:48:36 +00:00
|
|
|
#ifdef SMP
|
2003-12-12 07:33:51 +00:00
|
|
|
kseq->ksq_group->ksg_load++;
|
2004-02-01 02:48:36 +00:00
|
|
|
#else
|
|
|
|
kseq->ksq_sysload++;
|
2003-12-12 07:33:51 +00:00
|
|
|
#endif
|
2003-04-11 03:47:14 +00:00
|
|
|
if (ke->ke_ksegrp->kg_pri_class == PRI_TIMESHARE)
|
2004-06-16 00:26:31 +00:00
|
|
|
kseq_nice_add(kseq, ke->ke_proc->p_nice);
|
2003-02-03 05:30:07 +00:00
|
|
|
}
|
2003-04-11 03:47:14 +00:00
|
|
|
|
2003-04-03 00:29:28 +00:00
|
|
|
static void
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_rem(struct kseq *kseq, struct kse *ke)
|
2003-02-03 05:30:07 +00:00
|
|
|
{
|
2003-11-02 10:56:48 +00:00
|
|
|
int class;
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-11-02 10:56:48 +00:00
|
|
|
class = PRI_BASE(ke->ke_ksegrp->kg_pri_class);
|
|
|
|
if (class == PRI_TIMESHARE)
|
|
|
|
kseq->ksq_load_timeshare--;
|
2004-04-22 21:37:46 +00:00
|
|
|
if (class != PRI_ITHD && (ke->ke_proc->p_flag & P_NOLOAD) == 0)
|
2004-02-01 02:48:36 +00:00
|
|
|
#ifdef SMP
|
2003-12-12 07:33:51 +00:00
|
|
|
kseq->ksq_group->ksg_load--;
|
2004-02-01 02:48:36 +00:00
|
|
|
#else
|
|
|
|
kseq->ksq_sysload--;
|
2003-12-12 07:33:51 +00:00
|
|
|
#endif
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq->ksq_load--;
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR1(KTR_SCHED, "load: %d", kseq->ksq_load);
|
2003-04-11 03:47:14 +00:00
|
|
|
ke->ke_runq = NULL;
|
|
|
|
if (ke->ke_ksegrp->kg_pri_class == PRI_TIMESHARE)
|
2004-06-16 00:26:31 +00:00
|
|
|
kseq_nice_rem(kseq, ke->ke_proc->p_nice);
|
2003-02-03 05:30:07 +00:00
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
static void
|
|
|
|
kseq_nice_add(struct kseq *kseq, int nice)
|
2003-02-03 05:30:07 +00:00
|
|
|
{
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-04-11 03:47:14 +00:00
|
|
|
/* Normalize to zero. */
|
|
|
|
kseq->ksq_nice[nice + SCHED_PRI_NHALF]++;
|
2003-11-02 10:56:48 +00:00
|
|
|
if (nice < kseq->ksq_nicemin || kseq->ksq_load_timeshare == 1)
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq->ksq_nicemin = nice;
|
2003-02-03 05:30:07 +00:00
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
static void
|
|
|
|
kseq_nice_rem(struct kseq *kseq, int nice)
|
2003-02-03 05:30:07 +00:00
|
|
|
{
|
2003-04-11 03:47:14 +00:00
|
|
|
int n;
|
|
|
|
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-04-11 03:47:14 +00:00
|
|
|
/* Normalize to zero. */
|
|
|
|
n = nice + SCHED_PRI_NHALF;
|
|
|
|
kseq->ksq_nice[n]--;
|
|
|
|
KASSERT(kseq->ksq_nice[n] >= 0, ("Negative nice count."));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If this wasn't the smallest nice value or there are more in
|
|
|
|
* this bucket we can just return. Otherwise we have to recalculate
|
|
|
|
* the smallest nice.
|
|
|
|
*/
|
|
|
|
if (nice != kseq->ksq_nicemin ||
|
|
|
|
kseq->ksq_nice[n] != 0 ||
|
2003-11-02 10:56:48 +00:00
|
|
|
kseq->ksq_load_timeshare == 0)
|
2003-04-11 03:47:14 +00:00
|
|
|
return;
|
|
|
|
|
2003-11-02 03:49:32 +00:00
|
|
|
for (; n < SCHED_PRI_NRESV; n++)
|
2003-04-11 03:47:14 +00:00
|
|
|
if (kseq->ksq_nice[n]) {
|
|
|
|
kseq->ksq_nicemin = n - SCHED_PRI_NHALF;
|
|
|
|
return;
|
|
|
|
}
|
2003-02-03 05:30:07 +00:00
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
#ifdef SMP
|
2003-06-09 00:39:09 +00:00
|
|
|
/*
|
2003-11-15 07:32:07 +00:00
|
|
|
* sched_balance is a simple CPU load balancing algorithm. It operates by
|
2003-06-09 00:39:09 +00:00
|
|
|
* finding the least loaded and most loaded cpu and equalizing their load
|
|
|
|
* by migrating some processes.
|
|
|
|
*
|
|
|
|
* Dealing only with two CPUs at a time has two advantages. Firstly, most
|
|
|
|
* installations will only have 2 cpus. Secondly, load balancing too much at
|
|
|
|
* once can have an unpleasant effect on the system. The scheduler rarely has
|
|
|
|
* enough information to make perfect decisions. So this algorithm chooses
|
|
|
|
* algorithm simplicity and more gradual effects on load in larger systems.
|
|
|
|
*
|
|
|
|
* It could be improved by considering the priorities and slices assigned to
|
|
|
|
* each task prior to balancing them. There are many pathological cases with
|
|
|
|
* any approach and so the semi random algorithm below may work as well as any.
|
|
|
|
*
|
|
|
|
*/
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void
|
2004-06-02 05:46:48 +00:00
|
|
|
sched_balance(void)
|
2003-06-09 00:39:09 +00:00
|
|
|
{
|
2003-12-12 07:33:51 +00:00
|
|
|
struct kseq_group *high;
|
|
|
|
struct kseq_group *low;
|
|
|
|
struct kseq_group *ksg;
|
|
|
|
int cnt;
|
2003-06-09 00:39:09 +00:00
|
|
|
int i;
|
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
bal_tick = ticks + (random() % (hz * 2));
|
2003-06-28 08:24:42 +00:00
|
|
|
if (smp_started == 0)
|
2004-12-26 22:56:08 +00:00
|
|
|
return;
|
2003-12-12 07:33:51 +00:00
|
|
|
low = high = NULL;
|
|
|
|
i = random() % (ksg_maxid + 1);
|
|
|
|
for (cnt = 0; cnt <= ksg_maxid; cnt++) {
|
|
|
|
ksg = KSEQ_GROUP(i);
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
2003-12-12 07:33:51 +00:00
|
|
|
* Find the CPU with the highest load that has some
|
|
|
|
* threads to transfer.
|
2003-12-11 03:57:10 +00:00
|
|
|
*/
|
2003-12-12 07:33:51 +00:00
|
|
|
if ((high == NULL || ksg->ksg_load > high->ksg_load)
|
|
|
|
&& ksg->ksg_transferable)
|
|
|
|
high = ksg;
|
|
|
|
if (low == NULL || ksg->ksg_load < low->ksg_load)
|
|
|
|
low = ksg;
|
|
|
|
if (++i > ksg_maxid)
|
|
|
|
i = 0;
|
2003-06-09 00:39:09 +00:00
|
|
|
}
|
2003-12-12 07:33:51 +00:00
|
|
|
if (low != NULL && high != NULL && high != low)
|
|
|
|
sched_balance_pair(LIST_FIRST(&high->ksg_members),
|
|
|
|
LIST_FIRST(&low->ksg_members));
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
2004-06-02 05:46:48 +00:00
|
|
|
sched_balance_groups(void)
|
2003-12-12 07:33:51 +00:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
gbal_tick = ticks + (random() % (hz * 2));
|
2004-06-02 05:46:48 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-12-12 07:33:51 +00:00
|
|
|
if (smp_started)
|
|
|
|
for (i = 0; i <= ksg_maxid; i++)
|
|
|
|
sched_balance_group(KSEQ_GROUP(i));
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
sched_balance_group(struct kseq_group *ksg)
|
|
|
|
{
|
|
|
|
struct kseq *kseq;
|
|
|
|
struct kseq *high;
|
|
|
|
struct kseq *low;
|
|
|
|
int load;
|
|
|
|
|
|
|
|
if (ksg->ksg_transferable == 0)
|
|
|
|
return;
|
|
|
|
low = NULL;
|
|
|
|
high = NULL;
|
|
|
|
LIST_FOREACH(kseq, &ksg->ksg_members, ksq_siblings) {
|
|
|
|
load = kseq->ksq_load;
|
|
|
|
if (high == NULL || load > high->ksq_load)
|
|
|
|
high = kseq;
|
|
|
|
if (low == NULL || load < low->ksq_load)
|
|
|
|
low = kseq;
|
|
|
|
}
|
|
|
|
if (high != NULL && low != NULL && high != low)
|
|
|
|
sched_balance_pair(high, low);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
sched_balance_pair(struct kseq *high, struct kseq *low)
|
|
|
|
{
|
|
|
|
int transferable;
|
|
|
|
int high_load;
|
|
|
|
int low_load;
|
|
|
|
int move;
|
|
|
|
int diff;
|
|
|
|
int i;
|
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
|
|
|
* If we're transfering within a group we have to use this specific
|
|
|
|
* kseq's transferable count, otherwise we can steal from other members
|
|
|
|
* of the group.
|
|
|
|
*/
|
2003-12-12 07:33:51 +00:00
|
|
|
if (high->ksq_group == low->ksq_group) {
|
|
|
|
transferable = high->ksq_transferable;
|
|
|
|
high_load = high->ksq_load;
|
|
|
|
low_load = low->ksq_load;
|
|
|
|
} else {
|
|
|
|
transferable = high->ksq_group->ksg_transferable;
|
|
|
|
high_load = high->ksq_group->ksg_load;
|
|
|
|
low_load = low->ksq_group->ksg_load;
|
|
|
|
}
|
2003-12-11 03:57:10 +00:00
|
|
|
if (transferable == 0)
|
2003-12-12 07:33:51 +00:00
|
|
|
return;
|
2003-11-15 07:32:07 +00:00
|
|
|
/*
|
|
|
|
* Determine what the imbalance is and then adjust that to how many
|
2003-12-11 03:57:10 +00:00
|
|
|
* kses we actually have to give up (transferable).
|
2003-11-15 07:32:07 +00:00
|
|
|
*/
|
2003-12-12 07:33:51 +00:00
|
|
|
diff = high_load - low_load;
|
2003-06-09 00:39:09 +00:00
|
|
|
move = diff / 2;
|
|
|
|
if (diff & 0x1)
|
|
|
|
move++;
|
2003-12-11 03:57:10 +00:00
|
|
|
move = min(move, transferable);
|
2003-06-09 00:39:09 +00:00
|
|
|
for (i = 0; i < move; i++)
|
2003-12-12 07:33:51 +00:00
|
|
|
kseq_move(high, KSEQ_ID(low));
|
2003-06-09 00:39:09 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void
|
2003-06-09 00:39:09 +00:00
|
|
|
kseq_move(struct kseq *from, int cpu)
|
|
|
|
{
|
2003-12-11 03:57:10 +00:00
|
|
|
struct kseq *kseq;
|
|
|
|
struct kseq *to;
|
2003-06-09 00:39:09 +00:00
|
|
|
struct kse *ke;
|
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq = from;
|
|
|
|
to = KSEQ_CPU(cpu);
|
|
|
|
ke = kseq_steal(kseq, 1);
|
|
|
|
if (ke == NULL) {
|
|
|
|
struct kseq_group *ksg;
|
|
|
|
|
|
|
|
ksg = kseq->ksq_group;
|
|
|
|
LIST_FOREACH(kseq, &ksg->ksg_members, ksq_siblings) {
|
|
|
|
if (kseq == from || kseq->ksq_transferable == 0)
|
|
|
|
continue;
|
|
|
|
ke = kseq_steal(kseq, 1);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (ke == NULL)
|
|
|
|
panic("kseq_move: No KSEs available with a "
|
|
|
|
"transferable count of %d\n",
|
|
|
|
ksg->ksg_transferable);
|
|
|
|
}
|
|
|
|
if (kseq == to)
|
|
|
|
return;
|
2003-06-09 00:39:09 +00:00
|
|
|
ke->ke_state = KES_THREAD;
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq_runq_rem(kseq, ke);
|
|
|
|
kseq_load_rem(kseq, ke);
|
2003-11-03 03:27:22 +00:00
|
|
|
kseq_notify(ke, cpu);
|
2003-06-09 00:39:09 +00:00
|
|
|
}
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
static int
|
|
|
|
kseq_idled(struct kseq *kseq)
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
{
|
2003-12-11 03:57:10 +00:00
|
|
|
struct kseq_group *ksg;
|
|
|
|
struct kseq *steal;
|
|
|
|
struct kse *ke;
|
|
|
|
|
|
|
|
ksg = kseq->ksq_group;
|
|
|
|
/*
|
|
|
|
* If we're in a cpu group, try and steal kses from another cpu in
|
|
|
|
* the group before idling.
|
|
|
|
*/
|
|
|
|
if (ksg->ksg_cpus > 1 && ksg->ksg_transferable) {
|
|
|
|
LIST_FOREACH(steal, &ksg->ksg_members, ksq_siblings) {
|
|
|
|
if (steal == kseq || steal->ksq_transferable == 0)
|
|
|
|
continue;
|
|
|
|
ke = kseq_steal(steal, 0);
|
|
|
|
if (ke == NULL)
|
|
|
|
continue;
|
|
|
|
ke->ke_state = KES_THREAD;
|
|
|
|
kseq_runq_rem(steal, ke);
|
|
|
|
kseq_load_rem(steal, ke);
|
|
|
|
ke->ke_cpu = PCPU_GET(cpuid);
|
2004-12-26 22:56:08 +00:00
|
|
|
ke->ke_flags |= KEF_INTERNAL | KEF_HOLD;
|
|
|
|
sched_add(ke->ke_thread, SRQ_YIELDING);
|
2003-12-11 03:57:10 +00:00
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* We only set the idled bit when all of the cpus in the group are
|
|
|
|
* idle. Otherwise we could get into a situation where a KSE bounces
|
|
|
|
* back and forth between two idle cores on seperate physical CPUs.
|
|
|
|
*/
|
|
|
|
ksg->ksg_idlemask |= PCPU_GET(cpumask);
|
|
|
|
if (ksg->ksg_idlemask != ksg->ksg_cpumask)
|
|
|
|
return (1);
|
|
|
|
atomic_set_int(&kseq_idle, ksg->ksg_mask);
|
|
|
|
return (1);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
kseq_assign(struct kseq *kseq)
|
|
|
|
{
|
|
|
|
struct kse *nke;
|
|
|
|
struct kse *ke;
|
|
|
|
|
|
|
|
do {
|
2004-07-28 06:42:41 +00:00
|
|
|
*(volatile struct kse **)&ke = kseq->ksq_assigned;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
} while(!atomic_cmpset_ptr(&kseq->ksq_assigned, ke, NULL));
|
|
|
|
for (; ke != NULL; ke = nke) {
|
|
|
|
nke = ke->ke_assign;
|
2004-12-26 22:56:08 +00:00
|
|
|
kseq->ksq_group->ksg_load--;
|
|
|
|
kseq->ksq_load--;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
ke->ke_flags &= ~KEF_ASSIGNED;
|
2004-12-26 22:56:08 +00:00
|
|
|
ke->ke_flags |= KEF_INTERNAL | KEF_HOLD;
|
|
|
|
sched_add(ke->ke_thread, SRQ_YIELDING);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
kseq_notify(struct kse *ke, int cpu)
|
|
|
|
{
|
|
|
|
struct kseq *kseq;
|
|
|
|
struct thread *td;
|
|
|
|
struct pcpu *pcpu;
|
2004-12-26 22:56:08 +00:00
|
|
|
int class;
|
2004-08-10 07:52:21 +00:00
|
|
|
int prio;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
kseq = KSEQ_CPU(cpu);
|
|
|
|
/* XXX */
|
|
|
|
class = PRI_BASE(ke->ke_ksegrp->kg_pri_class);
|
|
|
|
if ((class == PRI_TIMESHARE || class == PRI_REALTIME) &&
|
|
|
|
(kseq_idle & kseq->ksq_group->ksg_mask))
|
|
|
|
atomic_clear_int(&kseq_idle, kseq->ksq_group->ksg_mask);
|
|
|
|
kseq->ksq_group->ksg_load++;
|
|
|
|
kseq->ksq_load++;
|
2003-12-14 02:06:29 +00:00
|
|
|
ke->ke_cpu = cpu;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
ke->ke_flags |= KEF_ASSIGNED;
|
2004-08-10 07:52:21 +00:00
|
|
|
prio = ke->ke_thread->td_priority;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Place a KSE on another cpu's queue and force a resched.
|
|
|
|
*/
|
|
|
|
do {
|
2004-07-28 06:42:41 +00:00
|
|
|
*(volatile struct kse **)&ke->ke_assign = kseq->ksq_assigned;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
} while(!atomic_cmpset_ptr(&kseq->ksq_assigned, ke->ke_assign, ke));
|
2004-08-10 07:52:21 +00:00
|
|
|
/*
|
|
|
|
* Without sched_lock we could lose a race where we set NEEDRESCHED
|
|
|
|
* on a thread that is switched out before the IPI is delivered. This
|
|
|
|
* would lead us to miss the resched. This will be a problem once
|
|
|
|
* sched_lock is pushed down.
|
|
|
|
*/
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
pcpu = pcpu_find(cpu);
|
|
|
|
td = pcpu->pc_curthread;
|
|
|
|
if (ke->ke_thread->td_priority < td->td_priority ||
|
|
|
|
td == pcpu->pc_idlethread) {
|
|
|
|
td->td_flags |= TDF_NEEDRESCHED;
|
|
|
|
ipi_selected(1 << cpu, IPI_AST);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct kse *
|
|
|
|
runq_steal(struct runq *rq)
|
|
|
|
{
|
|
|
|
struct rqhead *rqh;
|
|
|
|
struct rqbits *rqb;
|
|
|
|
struct kse *ke;
|
|
|
|
int word;
|
|
|
|
int bit;
|
|
|
|
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
rqb = &rq->rq_status;
|
|
|
|
for (word = 0; word < RQB_LEN; word++) {
|
|
|
|
if (rqb->rqb_bits[word] == 0)
|
|
|
|
continue;
|
|
|
|
for (bit = 0; bit < RQB_BPW; bit++) {
|
2003-12-07 09:57:51 +00:00
|
|
|
if ((rqb->rqb_bits[word] & (1ul << bit)) == 0)
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
continue;
|
|
|
|
rqh = &rq->rq_queues[bit + (word << RQB_L2BPW)];
|
|
|
|
TAILQ_FOREACH(ke, rqh, ke_procq) {
|
2004-12-26 22:56:08 +00:00
|
|
|
if (KSE_CAN_MIGRATE(ke))
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
return (ke);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return (NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct kse *
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq_steal(struct kseq *kseq, int stealidle)
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
{
|
|
|
|
struct kse *ke;
|
|
|
|
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
|
|
|
* Steal from next first to try to get a non-interactive task that
|
|
|
|
* may not have run for a while.
|
|
|
|
*/
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
if ((ke = runq_steal(kseq->ksq_next)) != NULL)
|
|
|
|
return (ke);
|
2003-12-11 03:57:10 +00:00
|
|
|
if ((ke = runq_steal(kseq->ksq_curr)) != NULL)
|
|
|
|
return (ke);
|
|
|
|
if (stealidle)
|
|
|
|
return (runq_steal(&kseq->ksq_idle));
|
|
|
|
return (NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
kseq_transfer(struct kseq *kseq, struct kse *ke, int class)
|
|
|
|
{
|
2004-12-26 22:56:08 +00:00
|
|
|
struct kseq_group *nksg;
|
2003-12-11 03:57:10 +00:00
|
|
|
struct kseq_group *ksg;
|
2004-12-26 22:56:08 +00:00
|
|
|
struct kseq *old;
|
2003-12-11 03:57:10 +00:00
|
|
|
int cpu;
|
2004-12-26 22:56:08 +00:00
|
|
|
int idx;
|
2003-12-11 03:57:10 +00:00
|
|
|
|
2003-12-20 14:03:14 +00:00
|
|
|
if (smp_started == 0)
|
|
|
|
return (0);
|
2003-12-11 03:57:10 +00:00
|
|
|
cpu = 0;
|
2003-12-20 14:03:14 +00:00
|
|
|
/*
|
2004-08-10 07:52:21 +00:00
|
|
|
* If our load exceeds a certain threshold we should attempt to
|
|
|
|
* reassign this thread. The first candidate is the cpu that
|
|
|
|
* originally ran the thread. If it is idle, assign it there,
|
|
|
|
* otherwise, pick an idle cpu.
|
|
|
|
*
|
|
|
|
* The threshold at which we start to reassign kses has a large impact
|
2003-12-20 14:03:14 +00:00
|
|
|
* on the overall performance of the system. Tuned too high and
|
|
|
|
* some CPUs may idle. Too low and there will be excess migration
|
2004-04-09 14:31:29 +00:00
|
|
|
* and context switches.
|
2003-12-20 14:03:14 +00:00
|
|
|
*/
|
2004-12-26 22:56:08 +00:00
|
|
|
old = KSEQ_CPU(ke->ke_cpu);
|
|
|
|
nksg = old->ksq_group;
|
2004-08-10 07:52:21 +00:00
|
|
|
ksg = kseq->ksq_group;
|
2004-12-26 22:56:08 +00:00
|
|
|
if (kseq_idle) {
|
|
|
|
if (kseq_idle & nksg->ksg_mask) {
|
|
|
|
cpu = ffs(nksg->ksg_idlemask);
|
|
|
|
if (cpu) {
|
|
|
|
CTR2(KTR_SCHED,
|
|
|
|
"kseq_transfer: %p found old cpu %X "
|
|
|
|
"in idlemask.", ke, cpu);
|
2004-08-10 07:52:21 +00:00
|
|
|
goto migrate;
|
2004-12-26 22:56:08 +00:00
|
|
|
}
|
2004-08-10 07:52:21 +00:00
|
|
|
}
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
|
|
|
* Multiple cpus could find this bit simultaneously
|
|
|
|
* but the race shouldn't be terrible.
|
|
|
|
*/
|
|
|
|
cpu = ffs(kseq_idle);
|
2004-12-26 22:56:08 +00:00
|
|
|
if (cpu) {
|
|
|
|
CTR2(KTR_SCHED, "kseq_transfer: %p found %X "
|
|
|
|
"in idlemask.", ke, cpu);
|
|
|
|
goto migrate;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
idx = 0;
|
|
|
|
#if 0
|
|
|
|
if (old->ksq_load < kseq->ksq_load) {
|
|
|
|
cpu = ke->ke_cpu + 1;
|
|
|
|
CTR2(KTR_SCHED, "kseq_transfer: %p old cpu %X "
|
|
|
|
"load less than ours.", ke, cpu);
|
|
|
|
goto migrate;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* No new CPU was found, look for one with less load.
|
|
|
|
*/
|
|
|
|
for (idx = 0; idx <= ksg_maxid; idx++) {
|
|
|
|
nksg = KSEQ_GROUP(idx);
|
|
|
|
if (nksg->ksg_load /*+ (nksg->ksg_cpus * 2)*/ < ksg->ksg_load) {
|
|
|
|
cpu = ffs(nksg->ksg_cpumask);
|
|
|
|
CTR2(KTR_SCHED, "kseq_transfer: %p cpu %X load less "
|
|
|
|
"than ours.", ke, cpu);
|
2004-08-10 07:52:21 +00:00
|
|
|
goto migrate;
|
2004-12-26 22:56:08 +00:00
|
|
|
}
|
2003-12-11 03:57:10 +00:00
|
|
|
}
|
2004-12-26 22:56:08 +00:00
|
|
|
#endif
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
|
|
|
* If another cpu in this group has idled, assign a thread over
|
|
|
|
* to them after checking to see if there are idled groups.
|
|
|
|
*/
|
2004-08-10 07:52:21 +00:00
|
|
|
if (ksg->ksg_idlemask) {
|
2003-12-11 03:57:10 +00:00
|
|
|
cpu = ffs(ksg->ksg_idlemask);
|
2004-12-26 22:56:08 +00:00
|
|
|
if (cpu) {
|
|
|
|
CTR2(KTR_SCHED, "kseq_transfer: %p cpu %X idle in "
|
|
|
|
"group.", ke, cpu);
|
2004-08-10 07:52:21 +00:00
|
|
|
goto migrate;
|
2004-12-26 22:56:08 +00:00
|
|
|
}
|
2003-12-11 03:57:10 +00:00
|
|
|
}
|
|
|
|
return (0);
|
2004-08-10 07:52:21 +00:00
|
|
|
migrate:
|
|
|
|
/*
|
|
|
|
* Now that we've found an idle CPU, migrate the thread.
|
|
|
|
*/
|
|
|
|
cpu--;
|
|
|
|
ke->ke_runq = NULL;
|
|
|
|
kseq_notify(ke, cpu);
|
|
|
|
|
|
|
|
return (1);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
}
|
2003-12-11 03:57:10 +00:00
|
|
|
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#endif /* SMP */
|
2003-02-03 05:30:07 +00:00
|
|
|
|
2003-07-08 06:19:40 +00:00
|
|
|
/*
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
* Pick the highest priority task we have and return it.
|
2003-07-08 06:19:40 +00:00
|
|
|
*/
|
|
|
|
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static struct kse *
|
|
|
|
kseq_choose(struct kseq *kseq)
|
2003-02-03 05:30:07 +00:00
|
|
|
{
|
|
|
|
struct runq *swap;
|
2004-10-30 12:19:15 +00:00
|
|
|
struct kse *ke;
|
|
|
|
int nice;
|
2003-02-03 05:30:07 +00:00
|
|
|
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-04-11 03:47:14 +00:00
|
|
|
swap = NULL;
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
ke = runq_choose(kseq->ksq_curr);
|
|
|
|
if (ke == NULL) {
|
|
|
|
/*
|
2004-07-02 19:09:50 +00:00
|
|
|
* We already swapped once and didn't get anywhere.
|
2003-04-11 03:47:14 +00:00
|
|
|
*/
|
|
|
|
if (swap)
|
|
|
|
break;
|
|
|
|
swap = kseq->ksq_curr;
|
|
|
|
kseq->ksq_curr = kseq->ksq_next;
|
|
|
|
kseq->ksq_next = swap;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* If we encounter a slice of 0 the kse is in a
|
|
|
|
* TIMESHARE kse group and its nice was too far out
|
|
|
|
* of the range that receives slices.
|
|
|
|
*/
|
2004-10-30 12:19:15 +00:00
|
|
|
nice = ke->ke_proc->p_nice + (0 - kseq->ksq_nicemin);
|
2004-12-14 10:34:27 +00:00
|
|
|
if (ke->ke_slice == 0 || (nice > SCHED_SLICE_NTHRESH &&
|
|
|
|
ke->ke_proc->p_nice != 0)) {
|
2003-04-11 03:47:14 +00:00
|
|
|
runq_remove(ke->ke_runq, ke);
|
|
|
|
sched_slice(ke);
|
|
|
|
ke->ke_runq = kseq->ksq_next;
|
2004-10-05 22:03:10 +00:00
|
|
|
runq_add(ke->ke_runq, ke, 0);
|
2003-04-11 03:47:14 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
return (ke);
|
2003-02-03 05:30:07 +00:00
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
return (runq_choose(&kseq->ksq_idle));
|
2003-04-02 06:46:43 +00:00
|
|
|
}
|
2003-01-29 07:00:51 +00:00
|
|
|
|
|
|
|
static void
|
|
|
|
kseq_setup(struct kseq *kseq)
|
|
|
|
{
|
2003-04-11 03:47:14 +00:00
|
|
|
runq_init(&kseq->ksq_timeshare[0]);
|
|
|
|
runq_init(&kseq->ksq_timeshare[1]);
|
2003-04-03 00:29:28 +00:00
|
|
|
runq_init(&kseq->ksq_idle);
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq->ksq_curr = &kseq->ksq_timeshare[0];
|
|
|
|
kseq->ksq_next = &kseq->ksq_timeshare[1];
|
2003-04-18 05:24:10 +00:00
|
|
|
kseq->ksq_load = 0;
|
2003-11-02 10:56:48 +00:00
|
|
|
kseq->ksq_load_timeshare = 0;
|
2003-01-29 07:00:51 +00:00
|
|
|
}
|
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
static void
|
|
|
|
sched_setup(void *dummy)
|
|
|
|
{
|
2003-07-07 21:08:28 +00:00
|
|
|
#ifdef SMP
|
2003-01-26 05:23:15 +00:00
|
|
|
int i;
|
2003-07-07 21:08:28 +00:00
|
|
|
#endif
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-06-28 06:04:47 +00:00
|
|
|
slice_min = (hz/100); /* 10ms */
|
|
|
|
slice_max = (hz/7); /* ~140ms */
|
2003-03-04 02:45:59 +00:00
|
|
|
|
2003-07-04 19:59:00 +00:00
|
|
|
#ifdef SMP
|
2003-12-12 07:33:51 +00:00
|
|
|
balance_groups = 0;
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
|
|
|
* Initialize the kseqs.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < MAXCPU; i++) {
|
|
|
|
struct kseq *ksq;
|
|
|
|
|
|
|
|
ksq = &kseq_cpu[i];
|
|
|
|
ksq->ksq_assigned = NULL;
|
|
|
|
kseq_setup(&kseq_cpu[i]);
|
|
|
|
}
|
2003-07-04 19:59:00 +00:00
|
|
|
if (smp_topology == NULL) {
|
2003-12-11 03:57:10 +00:00
|
|
|
struct kseq_group *ksg;
|
|
|
|
struct kseq *ksq;
|
2004-12-26 22:56:08 +00:00
|
|
|
int cpus;
|
2003-12-11 03:57:10 +00:00
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
for (cpus = 0, i = 0; i < MAXCPU; i++) {
|
|
|
|
if (CPU_ABSENT(i))
|
|
|
|
continue;
|
|
|
|
ksq = &kseq_cpu[cpus];
|
|
|
|
ksg = &kseq_groups[cpus];
|
2003-12-11 03:57:10 +00:00
|
|
|
/*
|
2004-06-02 05:46:48 +00:00
|
|
|
* Setup a kseq group with one member.
|
2003-12-11 03:57:10 +00:00
|
|
|
*/
|
|
|
|
ksq->ksq_transferable = 0;
|
|
|
|
ksq->ksq_group = ksg;
|
|
|
|
ksg->ksg_cpus = 1;
|
|
|
|
ksg->ksg_idlemask = 0;
|
|
|
|
ksg->ksg_cpumask = ksg->ksg_mask = 1 << i;
|
2003-12-12 07:33:51 +00:00
|
|
|
ksg->ksg_load = 0;
|
2003-12-11 03:57:10 +00:00
|
|
|
ksg->ksg_transferable = 0;
|
|
|
|
LIST_INIT(&ksg->ksg_members);
|
|
|
|
LIST_INSERT_HEAD(&ksg->ksg_members, ksq, ksq_siblings);
|
2004-12-26 22:56:08 +00:00
|
|
|
cpus++;
|
2003-07-04 19:59:00 +00:00
|
|
|
}
|
2004-12-26 22:56:08 +00:00
|
|
|
ksg_maxid = cpus - 1;
|
2003-07-04 19:59:00 +00:00
|
|
|
} else {
|
2003-12-11 03:57:10 +00:00
|
|
|
struct kseq_group *ksg;
|
|
|
|
struct cpu_group *cg;
|
2003-07-04 19:59:00 +00:00
|
|
|
int j;
|
2003-04-11 03:47:14 +00:00
|
|
|
|
2003-07-04 19:59:00 +00:00
|
|
|
for (i = 0; i < smp_topology->ct_count; i++) {
|
|
|
|
cg = &smp_topology->ct_group[i];
|
2003-12-11 03:57:10 +00:00
|
|
|
ksg = &kseq_groups[i];
|
|
|
|
/*
|
|
|
|
* Initialize the group.
|
|
|
|
*/
|
|
|
|
ksg->ksg_idlemask = 0;
|
2003-12-12 07:33:51 +00:00
|
|
|
ksg->ksg_load = 0;
|
2003-12-11 03:57:10 +00:00
|
|
|
ksg->ksg_transferable = 0;
|
|
|
|
ksg->ksg_cpus = cg->cg_count;
|
|
|
|
ksg->ksg_cpumask = cg->cg_mask;
|
|
|
|
LIST_INIT(&ksg->ksg_members);
|
|
|
|
/*
|
|
|
|
* Find all of the group members and add them.
|
|
|
|
*/
|
|
|
|
for (j = 0; j < MAXCPU; j++) {
|
|
|
|
if ((cg->cg_mask & (1 << j)) != 0) {
|
|
|
|
if (ksg->ksg_mask == 0)
|
|
|
|
ksg->ksg_mask = 1 << j;
|
|
|
|
kseq_cpu[j].ksq_transferable = 0;
|
|
|
|
kseq_cpu[j].ksq_group = ksg;
|
|
|
|
LIST_INSERT_HEAD(&ksg->ksg_members,
|
|
|
|
&kseq_cpu[j], ksq_siblings);
|
|
|
|
}
|
|
|
|
}
|
2003-12-12 07:33:51 +00:00
|
|
|
if (ksg->ksg_cpus > 1)
|
|
|
|
balance_groups = 1;
|
2003-07-04 19:59:00 +00:00
|
|
|
}
|
2003-12-12 07:33:51 +00:00
|
|
|
ksg_maxid = smp_topology->ct_count - 1;
|
2003-07-04 19:59:00 +00:00
|
|
|
}
|
2003-12-12 07:33:51 +00:00
|
|
|
/*
|
|
|
|
* Stagger the group and global load balancer so they do not
|
|
|
|
* interfere with each other.
|
|
|
|
*/
|
2004-06-02 05:46:48 +00:00
|
|
|
bal_tick = ticks + hz;
|
2003-12-12 07:33:51 +00:00
|
|
|
if (balance_groups)
|
2004-06-02 05:46:48 +00:00
|
|
|
gbal_tick = ticks + (hz / 2);
|
2003-07-04 19:59:00 +00:00
|
|
|
#else
|
|
|
|
kseq_setup(KSEQ_SELF());
|
2003-06-09 00:39:09 +00:00
|
|
|
#endif
|
2003-07-04 19:59:00 +00:00
|
|
|
mtx_lock_spin(&sched_lock);
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_add(KSEQ_SELF(), &kse0);
|
2003-07-04 19:59:00 +00:00
|
|
|
mtx_unlock_spin(&sched_lock);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Scale the scheduling priority according to the "interactivity" of this
|
|
|
|
* process.
|
|
|
|
*/
|
2003-04-11 03:47:14 +00:00
|
|
|
static void
|
2003-01-26 05:23:15 +00:00
|
|
|
sched_priority(struct ksegrp *kg)
|
|
|
|
{
|
|
|
|
int pri;
|
|
|
|
|
|
|
|
if (kg->kg_pri_class != PRI_TIMESHARE)
|
2003-04-11 03:47:14 +00:00
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
pri = SCHED_PRI_INTERACT(sched_interact_score(kg));
|
2003-03-04 02:45:59 +00:00
|
|
|
pri += SCHED_PRI_BASE;
|
2004-06-16 00:26:31 +00:00
|
|
|
pri += kg->kg_proc->p_nice;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
if (pri > PRI_MAX_TIMESHARE)
|
|
|
|
pri = PRI_MAX_TIMESHARE;
|
|
|
|
else if (pri < PRI_MIN_TIMESHARE)
|
|
|
|
pri = PRI_MIN_TIMESHARE;
|
|
|
|
|
|
|
|
kg->kg_user_pri = pri;
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2003-04-02 06:46:43 +00:00
|
|
|
* Calculate a time slice based on the properties of the kseg and the runq
|
2003-04-03 00:29:28 +00:00
|
|
|
* that we're on. This is only for PRI_TIMESHARE ksegrps.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-04-02 06:46:43 +00:00
|
|
|
static void
|
|
|
|
sched_slice(struct kse *ke)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2003-04-11 03:47:14 +00:00
|
|
|
struct kseq *kseq;
|
2003-04-02 06:46:43 +00:00
|
|
|
struct ksegrp *kg;
|
|
|
|
|
|
|
|
kg = ke->ke_ksegrp;
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq = KSEQ_CPU(ke->ke_cpu);
|
2003-04-02 06:46:43 +00:00
|
|
|
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
if (ke->ke_thread->td_flags & TDF_BORROWING) {
|
2004-12-14 10:34:27 +00:00
|
|
|
ke->ke_slice = SCHED_SLICE_MIN;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2003-04-02 06:46:43 +00:00
|
|
|
/*
|
|
|
|
* Rationale:
|
2004-08-10 07:52:21 +00:00
|
|
|
* KSEs in interactive ksegs get a minimal slice so that we
|
2003-04-02 06:46:43 +00:00
|
|
|
* quickly notice if it abuses its advantage.
|
|
|
|
*
|
|
|
|
* KSEs in non-interactive ksegs are assigned a slice that is
|
|
|
|
* based on the ksegs nice value relative to the least nice kseg
|
|
|
|
* on the run queue for this cpu.
|
|
|
|
*
|
|
|
|
* If the KSE is less nice than all others it gets the maximum
|
|
|
|
* slice and other KSEs will adjust their slice relative to
|
|
|
|
* this when they first expire.
|
|
|
|
*
|
|
|
|
* There is 20 point window that starts relative to the least
|
|
|
|
* nice kse on the run queue. Slice size is determined by
|
|
|
|
* the kse distance from the last nice ksegrp.
|
|
|
|
*
|
2003-11-02 04:10:15 +00:00
|
|
|
* If the kse is outside of the window it will get no slice
|
|
|
|
* and will be reevaluated each time it is selected on the
|
|
|
|
* run queue. The exception to this is nice 0 ksegs when
|
|
|
|
* a nice -20 is running. They are always granted a minimum
|
|
|
|
* slice.
|
2003-04-02 06:46:43 +00:00
|
|
|
*/
|
2003-04-11 03:47:14 +00:00
|
|
|
if (!SCHED_INTERACTIVE(kg)) {
|
2003-04-02 06:46:43 +00:00
|
|
|
int nice;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-06-16 00:26:31 +00:00
|
|
|
nice = kg->kg_proc->p_nice + (0 - kseq->ksq_nicemin);
|
2003-11-02 10:56:48 +00:00
|
|
|
if (kseq->ksq_load_timeshare == 0 ||
|
2004-06-16 00:26:31 +00:00
|
|
|
kg->kg_proc->p_nice < kseq->ksq_nicemin)
|
2003-04-02 06:46:43 +00:00
|
|
|
ke->ke_slice = SCHED_SLICE_MAX;
|
2003-11-02 04:10:15 +00:00
|
|
|
else if (nice <= SCHED_SLICE_NTHRESH)
|
2003-04-02 06:46:43 +00:00
|
|
|
ke->ke_slice = SCHED_SLICE_NICE(nice);
|
2004-06-16 00:26:31 +00:00
|
|
|
else if (kg->kg_proc->p_nice == 0)
|
2003-11-02 04:10:15 +00:00
|
|
|
ke->ke_slice = SCHED_SLICE_MIN;
|
2003-04-02 06:46:43 +00:00
|
|
|
else
|
|
|
|
ke->ke_slice = 0;
|
|
|
|
} else
|
2003-12-20 12:54:35 +00:00
|
|
|
ke->ke_slice = SCHED_SLICE_INTERACTIVE;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-04-02 06:46:43 +00:00
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
2003-11-02 03:36:33 +00:00
|
|
|
/*
|
|
|
|
* This routine enforces a maximum limit on the amount of scheduling history
|
|
|
|
* kept. It is called after either the slptime or runtime is adjusted.
|
|
|
|
* This routine will not operate correctly when slp or run times have been
|
|
|
|
* adjusted to more than double their maximum.
|
|
|
|
*/
|
2003-06-17 06:39:51 +00:00
|
|
|
static void
|
|
|
|
sched_interact_update(struct ksegrp *kg)
|
|
|
|
{
|
2003-11-02 03:36:33 +00:00
|
|
|
int sum;
|
|
|
|
|
|
|
|
sum = kg->kg_runtime + kg->kg_slptime;
|
|
|
|
if (sum < SCHED_SLP_RUN_MAX)
|
|
|
|
return;
|
|
|
|
/*
|
|
|
|
* If we have exceeded by more than 1/5th then the algorithm below
|
|
|
|
* will not bring us back into range. Dividing by two here forces
|
2004-08-10 07:52:21 +00:00
|
|
|
* us into the range of [4/5 * SCHED_INTERACT_MAX, SCHED_INTERACT_MAX]
|
2003-11-02 03:36:33 +00:00
|
|
|
*/
|
2004-04-04 19:12:56 +00:00
|
|
|
if (sum > (SCHED_SLP_RUN_MAX / 5) * 6) {
|
2003-11-02 03:36:33 +00:00
|
|
|
kg->kg_runtime /= 2;
|
|
|
|
kg->kg_slptime /= 2;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
kg->kg_runtime = (kg->kg_runtime / 5) * 4;
|
|
|
|
kg->kg_slptime = (kg->kg_slptime / 5) * 4;
|
|
|
|
}
|
2003-10-27 06:47:05 +00:00
|
|
|
|
2003-11-02 03:36:33 +00:00
|
|
|
static void
|
|
|
|
sched_interact_fork(struct ksegrp *kg)
|
|
|
|
{
|
|
|
|
int ratio;
|
|
|
|
int sum;
|
|
|
|
|
|
|
|
sum = kg->kg_runtime + kg->kg_slptime;
|
|
|
|
if (sum > SCHED_SLP_RUN_FORK) {
|
|
|
|
ratio = sum / SCHED_SLP_RUN_FORK;
|
|
|
|
kg->kg_runtime /= ratio;
|
|
|
|
kg->kg_slptime /= ratio;
|
2003-06-17 06:39:51 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2003-03-04 02:45:59 +00:00
|
|
|
static int
|
|
|
|
sched_interact_score(struct ksegrp *kg)
|
|
|
|
{
|
2003-06-15 02:18:29 +00:00
|
|
|
int div;
|
2003-03-04 02:45:59 +00:00
|
|
|
|
|
|
|
if (kg->kg_runtime > kg->kg_slptime) {
|
2003-06-15 02:18:29 +00:00
|
|
|
div = max(1, kg->kg_runtime / SCHED_INTERACT_HALF);
|
|
|
|
return (SCHED_INTERACT_HALF +
|
|
|
|
(SCHED_INTERACT_HALF - (kg->kg_slptime / div)));
|
|
|
|
} if (kg->kg_slptime > kg->kg_runtime) {
|
|
|
|
div = max(1, kg->kg_slptime / SCHED_INTERACT_HALF);
|
|
|
|
return (kg->kg_runtime / div);
|
2003-03-04 02:45:59 +00:00
|
|
|
}
|
|
|
|
|
2003-06-15 02:18:29 +00:00
|
|
|
/*
|
|
|
|
* This can happen if slptime and runtime are 0.
|
|
|
|
*/
|
|
|
|
return (0);
|
2003-03-04 02:45:59 +00:00
|
|
|
|
|
|
|
}
|
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
/*
|
|
|
|
* Very early in the boot some setup of scheduler-specific
|
|
|
|
* parts of proc0 and of soem scheduler resources needs to be done.
|
|
|
|
* Called from:
|
|
|
|
* proc0_init()
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
schedinit(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Set up the scheduler specific parts of proc0.
|
|
|
|
*/
|
|
|
|
proc0.p_sched = NULL; /* XXX */
|
2004-10-05 21:10:44 +00:00
|
|
|
ksegrp0.kg_sched = &kg_sched0;
|
|
|
|
thread0.td_sched = &kse0;
|
2004-09-05 02:09:54 +00:00
|
|
|
kse0.ke_thread = &thread0;
|
|
|
|
kse0.ke_state = KES_THREAD;
|
|
|
|
kg_sched0.skg_concurrency = 1;
|
|
|
|
kg_sched0.skg_avail_opennings = 0; /* we are already running */
|
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
/*
|
|
|
|
* This is only somewhat accurate since given many processes of the same
|
|
|
|
* priority they will switch when their slices run out, which will be
|
|
|
|
* at most SCHED_SLICE_MAX.
|
|
|
|
*/
|
2003-01-26 05:23:15 +00:00
|
|
|
int
|
|
|
|
sched_rr_interval(void)
|
|
|
|
{
|
|
|
|
return (SCHED_SLICE_MAX);
|
|
|
|
}
|
|
|
|
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
static void
|
2003-01-26 05:23:15 +00:00
|
|
|
sched_pctcpu_update(struct kse *ke)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Adjust counters and watermark for pctcpu calc.
|
2003-06-15 02:18:29 +00:00
|
|
|
*/
|
2003-09-20 02:05:58 +00:00
|
|
|
if (ke->ke_ltick > ticks - SCHED_CPU_TICKS) {
|
|
|
|
/*
|
|
|
|
* Shift the tick count out so that the divide doesn't
|
|
|
|
* round away our results.
|
|
|
|
*/
|
|
|
|
ke->ke_ticks <<= 10;
|
|
|
|
ke->ke_ticks = (ke->ke_ticks / (ticks - ke->ke_ftick)) *
|
|
|
|
SCHED_CPU_TICKS;
|
|
|
|
ke->ke_ticks >>= 10;
|
|
|
|
} else
|
|
|
|
ke->ke_ticks = 0;
|
2003-01-26 05:23:15 +00:00
|
|
|
ke->ke_ltick = ticks;
|
|
|
|
ke->ke_ftick = ke->ke_ltick - SCHED_CPU_TICKS;
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
sched_thread_priority(struct thread *td, u_char prio)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2003-10-27 06:47:05 +00:00
|
|
|
struct kse *ke;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR6(KTR_SCHED, "sched_prio: %p(%s) prio %d newprio %d by %p(%s)",
|
|
|
|
td, td->td_proc->p_comm, td->td_priority, prio, curthread,
|
|
|
|
curthread->td_proc->p_comm);
|
2003-10-27 06:47:05 +00:00
|
|
|
ke = td->td_kse;
|
2003-01-26 05:23:15 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
if (td->td_priority == prio)
|
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
if (TD_ON_RUNQ(td)) {
|
2003-10-27 06:47:05 +00:00
|
|
|
/*
|
|
|
|
* If the priority has been elevated due to priority
|
|
|
|
* propagation, we may have to move ourselves to a new
|
|
|
|
* queue. We still call adjustrunqueue below in case kse
|
|
|
|
* needs to fix things up.
|
|
|
|
*/
|
2004-12-14 10:34:27 +00:00
|
|
|
if (prio < td->td_priority && ke->ke_runq != NULL &&
|
2003-11-02 04:25:59 +00:00
|
|
|
(ke->ke_flags & KEF_ASSIGNED) == 0 &&
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
ke->ke_runq != KSEQ_CPU(ke->ke_cpu)->ksq_curr) {
|
2003-10-27 06:47:05 +00:00
|
|
|
runq_remove(ke->ke_runq, ke);
|
|
|
|
ke->ke_runq = KSEQ_CPU(ke->ke_cpu)->ksq_curr;
|
2004-10-05 22:03:10 +00:00
|
|
|
runq_add(ke->ke_runq, ke, 0);
|
2003-10-27 06:47:05 +00:00
|
|
|
}
|
2004-08-12 07:56:33 +00:00
|
|
|
/*
|
|
|
|
* Hold this kse on this cpu so that sched_prio() doesn't
|
|
|
|
* cause excessive migration. We only want migration to
|
|
|
|
* happen as the result of a wakeup.
|
|
|
|
*/
|
|
|
|
ke->ke_flags |= KEF_HOLD;
|
2003-08-26 11:33:15 +00:00
|
|
|
adjustrunqueue(td, prio);
|
2004-12-26 22:56:08 +00:00
|
|
|
ke->ke_flags &= ~KEF_HOLD;
|
2003-10-27 06:47:05 +00:00
|
|
|
} else
|
2003-08-26 11:33:15 +00:00
|
|
|
td->td_priority = prio;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
/*
|
|
|
|
* Update a thread's priority when it is lent another thread's
|
|
|
|
* priority.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
sched_lend_prio(struct thread *td, u_char prio)
|
|
|
|
{
|
|
|
|
|
|
|
|
td->td_flags |= TDF_BORROWING;
|
|
|
|
sched_thread_priority(td, prio);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Restore a thread's priority when priority propagation is
|
|
|
|
* over. The prio argument is the minimum priority the thread
|
|
|
|
* needs to have to satisfy other possible priority lending
|
|
|
|
* requests. If the thread's regular priority is less
|
|
|
|
* important than prio, the thread will keep a priority boost
|
|
|
|
* of prio.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
sched_unlend_prio(struct thread *td, u_char prio)
|
|
|
|
{
|
|
|
|
u_char base_pri;
|
|
|
|
|
|
|
|
if (td->td_base_pri >= PRI_MIN_TIMESHARE &&
|
|
|
|
td->td_base_pri <= PRI_MAX_TIMESHARE)
|
|
|
|
base_pri = td->td_ksegrp->kg_user_pri;
|
|
|
|
else
|
|
|
|
base_pri = td->td_base_pri;
|
|
|
|
if (prio >= base_pri) {
|
2004-12-30 22:17:00 +00:00
|
|
|
td->td_flags &= ~TDF_BORROWING;
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
sched_thread_priority(td, base_pri);
|
|
|
|
} else
|
|
|
|
sched_lend_prio(td, prio);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
sched_prio(struct thread *td, u_char prio)
|
|
|
|
{
|
|
|
|
u_char oldprio;
|
|
|
|
|
|
|
|
/* First, update the base priority. */
|
|
|
|
td->td_base_pri = prio;
|
|
|
|
|
|
|
|
/*
|
2004-12-30 22:17:00 +00:00
|
|
|
* If the thread is borrowing another thread's priority, don't
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
* ever lower the priority.
|
|
|
|
*/
|
|
|
|
if (td->td_flags & TDF_BORROWING && td->td_priority < prio)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Change the real priority. */
|
|
|
|
oldprio = td->td_priority;
|
|
|
|
sched_thread_priority(td, prio);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the thread is on a turnstile, then let the turnstile update
|
|
|
|
* its state.
|
|
|
|
*/
|
|
|
|
if (TD_ON_LOCK(td) && oldprio != prio)
|
|
|
|
turnstile_adjust(td, oldprio);
|
|
|
|
}
|
2004-12-30 22:17:00 +00:00
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
void
|
2004-09-10 21:04:38 +00:00
|
|
|
sched_switch(struct thread *td, struct thread *newtd, int flags)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2004-12-26 22:56:08 +00:00
|
|
|
struct kseq *ksq;
|
2003-01-26 05:23:15 +00:00
|
|
|
struct kse *ke;
|
|
|
|
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
|
|
|
|
ke = td->td_kse;
|
2004-12-26 22:56:08 +00:00
|
|
|
ksq = KSEQ_SELF();
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-08-12 07:56:33 +00:00
|
|
|
td->td_lastcpu = td->td_oncpu;
|
2003-04-10 17:35:44 +00:00
|
|
|
td->td_oncpu = NOCPU;
|
2004-07-16 21:04:55 +00:00
|
|
|
td->td_flags &= ~TDF_NEEDRESCHED;
|
|
|
|
td->td_pflags &= ~TDP_OWEPREEMPT;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-12-11 04:00:49 +00:00
|
|
|
/*
|
|
|
|
* If the KSE has been assigned it may be in the process of switching
|
|
|
|
* to the new cpu. This is the case in sched_bind().
|
|
|
|
*/
|
2004-12-26 22:56:08 +00:00
|
|
|
if (td == PCPU_GET(idlethread)) {
|
|
|
|
TD_SET_CAN_RUN(td);
|
|
|
|
} else if ((ke->ke_flags & KEF_ASSIGNED) == 0) {
|
|
|
|
/* We are ending our run so make our slot available again */
|
|
|
|
SLOT_RELEASE(td->td_ksegrp);
|
|
|
|
if (ke->ke_runq == NULL)
|
|
|
|
panic("Thread not on runq.");
|
|
|
|
kseq_load_rem(ksq, ke);
|
|
|
|
if (TD_IS_RUNNING(td)) {
|
|
|
|
/*
|
|
|
|
* Don't allow the thread to migrate
|
|
|
|
* from a preemption.
|
|
|
|
*/
|
|
|
|
ke->ke_flags |= KEF_HOLD;
|
|
|
|
setrunqueue(td, (flags & SW_PREEMPT) ?
|
|
|
|
SRQ_OURSELF|SRQ_YIELDING|SRQ_PREEMPTED :
|
|
|
|
SRQ_OURSELF|SRQ_YIELDING);
|
|
|
|
ke->ke_flags &= ~KEF_HOLD;
|
|
|
|
} else if ((td->td_proc->p_flag & P_HADTHREADS) &&
|
|
|
|
(newtd == NULL || newtd->td_ksegrp != td->td_ksegrp))
|
|
|
|
/*
|
|
|
|
* We will not be on the run queue.
|
|
|
|
* So we must be sleeping or similar.
|
|
|
|
* Don't use the slot if we will need it
|
|
|
|
* for newtd.
|
|
|
|
*/
|
|
|
|
slot_fill(td->td_ksegrp);
|
2003-10-16 20:32:57 +00:00
|
|
|
}
|
2004-10-05 21:10:44 +00:00
|
|
|
if (newtd != NULL) {
|
2004-10-05 22:03:10 +00:00
|
|
|
/*
|
|
|
|
* If we bring in a thread,
|
|
|
|
* then account for it as if it had been added to the
|
|
|
|
* run queue and then chosen.
|
|
|
|
*/
|
2004-10-05 21:26:27 +00:00
|
|
|
newtd->td_kse->ke_flags |= KEF_DIDRUN;
|
2004-12-26 22:56:08 +00:00
|
|
|
newtd->td_kse->ke_runq = ksq->ksq_curr;
|
2004-10-05 21:10:44 +00:00
|
|
|
SLOT_USE(newtd->td_ksegrp);
|
2004-10-05 22:14:02 +00:00
|
|
|
TD_SET_RUNNING(newtd);
|
2004-07-02 19:09:50 +00:00
|
|
|
kseq_load_add(KSEQ_SELF(), newtd->td_kse);
|
2004-10-05 21:10:44 +00:00
|
|
|
} else
|
2004-08-10 07:52:21 +00:00
|
|
|
newtd = choosethread();
|
2003-10-16 08:53:46 +00:00
|
|
|
if (td != newtd)
|
|
|
|
cpu_switch(td, newtd);
|
|
|
|
sched_lock.mtx_lock = (uintptr_t)td;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-04-10 17:35:44 +00:00
|
|
|
td->td_oncpu = PCPU_GET(cpuid);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2004-06-16 00:26:31 +00:00
|
|
|
sched_nice(struct proc *p, int nice)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2004-06-16 00:26:31 +00:00
|
|
|
struct ksegrp *kg;
|
2003-04-11 03:47:14 +00:00
|
|
|
struct kse *ke;
|
2003-01-26 05:23:15 +00:00
|
|
|
struct thread *td;
|
2003-04-11 03:47:14 +00:00
|
|
|
struct kseq *kseq;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-06-16 00:26:31 +00:00
|
|
|
PROC_LOCK_ASSERT(p, MA_OWNED);
|
2003-04-22 20:50:38 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-04-11 03:47:14 +00:00
|
|
|
/*
|
|
|
|
* We need to adjust the nice counts for running KSEs.
|
|
|
|
*/
|
2004-06-16 00:26:31 +00:00
|
|
|
FOREACH_KSEGRP_IN_PROC(p, kg) {
|
|
|
|
if (kg->kg_pri_class == PRI_TIMESHARE) {
|
2004-09-05 02:09:54 +00:00
|
|
|
FOREACH_THREAD_IN_GROUP(kg, td) {
|
|
|
|
ke = td->td_kse;
|
2004-06-16 00:26:31 +00:00
|
|
|
if (ke->ke_runq == NULL)
|
|
|
|
continue;
|
|
|
|
kseq = KSEQ_CPU(ke->ke_cpu);
|
|
|
|
kseq_nice_rem(kseq, p->p_nice);
|
|
|
|
kseq_nice_add(kseq, nice);
|
|
|
|
}
|
2003-04-11 03:47:14 +00:00
|
|
|
}
|
2004-06-16 00:26:31 +00:00
|
|
|
}
|
|
|
|
p->p_nice = nice;
|
|
|
|
FOREACH_KSEGRP_IN_PROC(p, kg) {
|
|
|
|
sched_priority(kg);
|
|
|
|
FOREACH_THREAD_IN_GROUP(kg, td)
|
|
|
|
td->td_flags |= TDF_NEEDRESCHED;
|
|
|
|
}
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
Switch the sleep/wakeup and condition variable implementations to use the
sleep queue interface:
- Sleep queues attempt to merge some of the benefits of both sleep queues
and condition variables. Having sleep qeueus in a hash table avoids
having to allocate a queue head for each wait channel. Thus, struct cv
has shrunk down to just a single char * pointer now. However, the
hash table does not hold threads directly, but queue heads. This means
that once you have located a queue in the hash bucket, you no longer have
to walk the rest of the hash chain looking for threads. Instead, you have
a list of all the threads sleeping on that wait channel.
- Outside of the sleepq code and the sleep/cv code the kernel no longer
differentiates between cv's and sleep/wakeup. For example, calls to
abortsleep() and cv_abort() are replaced with a call to sleepq_abort().
Thus, the TDF_CVWAITQ flag is removed. Also, calls to unsleep() and
cv_waitq_remove() have been replaced with calls to sleepq_remove().
- The sched_sleep() function no longer accepts a priority argument as
sleep's no longer inherently bump the priority. Instead, this is soley
a propery of msleep() which explicitly calls sched_prio() before
blocking.
- The TDF_ONSLEEPQ flag has been dropped as it was never used. The
associated TDF_SET_ONSLEEPQ and TDF_CLR_ON_SLEEPQ macros have also been
dropped and replaced with a single explicit clearing of td_wchan.
TD_SET_ONSLEEPQ() would really have only made sense if it had taken
the wait channel and message as arguments anyway. Now that that only
happens in one place, a macro would be overkill.
2004-02-27 18:52:44 +00:00
|
|
|
sched_sleep(struct thread *td)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
|
|
|
|
td->td_slptime = ticks;
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
sched_wakeup(struct thread *td)
|
|
|
|
{
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Let the kseg know how long we slept for. This is because process
|
|
|
|
* interactivity behavior is modeled in the kseg.
|
|
|
|
*/
|
|
|
|
if (td->td_slptime) {
|
2003-03-03 04:11:40 +00:00
|
|
|
struct ksegrp *kg;
|
2003-04-11 03:47:14 +00:00
|
|
|
int hzticks;
|
2003-03-03 04:11:40 +00:00
|
|
|
|
|
|
|
kg = td->td_ksegrp;
|
2003-11-02 03:36:33 +00:00
|
|
|
hzticks = (ticks - td->td_slptime) << 10;
|
|
|
|
if (hzticks >= SCHED_SLP_RUN_MAX) {
|
|
|
|
kg->kg_slptime = SCHED_SLP_RUN_MAX;
|
|
|
|
kg->kg_runtime = 1;
|
|
|
|
} else {
|
|
|
|
kg->kg_slptime += hzticks;
|
|
|
|
sched_interact_update(kg);
|
|
|
|
}
|
2003-03-03 04:11:40 +00:00
|
|
|
sched_priority(kg);
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_slice(td->td_kse);
|
2003-03-03 04:11:40 +00:00
|
|
|
td->td_slptime = 0;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
2004-09-01 02:11:28 +00:00
|
|
|
setrunqueue(td, SRQ_BORING);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Penalize the parent for creating a new child and initialize the child's
|
|
|
|
* priority.
|
|
|
|
*/
|
|
|
|
void
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_fork(struct thread *td, struct thread *childtd)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_fork_ksegrp(td, childtd->td_ksegrp);
|
|
|
|
sched_fork_thread(td, childtd);
|
2003-04-11 03:47:14 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2004-07-18 23:36:13 +00:00
|
|
|
sched_fork_ksegrp(struct thread *td, struct ksegrp *child)
|
2003-04-11 03:47:14 +00:00
|
|
|
{
|
2004-07-18 23:36:13 +00:00
|
|
|
struct ksegrp *kg = td->td_ksegrp;
|
2004-09-05 02:09:54 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-06-15 02:18:29 +00:00
|
|
|
|
2003-11-02 03:36:33 +00:00
|
|
|
child->kg_slptime = kg->kg_slptime;
|
|
|
|
child->kg_runtime = kg->kg_runtime;
|
|
|
|
child->kg_user_pri = kg->kg_user_pri;
|
|
|
|
sched_interact_fork(child);
|
2003-06-17 06:39:51 +00:00
|
|
|
kg->kg_runtime += tickincr << 10;
|
|
|
|
sched_interact_update(kg);
|
2003-04-11 03:47:14 +00:00
|
|
|
}
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
void
|
|
|
|
sched_fork_thread(struct thread *td, struct thread *child)
|
|
|
|
{
|
2004-09-05 02:09:54 +00:00
|
|
|
struct kse *ke;
|
|
|
|
struct kse *ke2;
|
|
|
|
|
|
|
|
sched_newthread(child);
|
|
|
|
ke = td->td_kse;
|
|
|
|
ke2 = child->td_kse;
|
|
|
|
ke2->ke_slice = 1; /* Attempt to quickly learn interactivity. */
|
|
|
|
ke2->ke_cpu = ke->ke_cpu;
|
|
|
|
ke2->ke_runq = NULL;
|
|
|
|
|
|
|
|
/* Grab our parents cpu estimation information. */
|
|
|
|
ke2->ke_ticks = ke->ke_ticks;
|
|
|
|
ke2->ke_ltick = ke->ke_ltick;
|
|
|
|
ke2->ke_ftick = ke->ke_ftick;
|
2003-04-11 03:47:14 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
sched_class(struct ksegrp *kg, int class)
|
|
|
|
{
|
|
|
|
struct kseq *kseq;
|
|
|
|
struct kse *ke;
|
2004-09-05 02:09:54 +00:00
|
|
|
struct thread *td;
|
2003-11-02 10:56:48 +00:00
|
|
|
int nclass;
|
|
|
|
int oclass;
|
2003-04-11 03:47:14 +00:00
|
|
|
|
2003-04-23 18:51:05 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-04-11 03:47:14 +00:00
|
|
|
if (kg->kg_pri_class == class)
|
|
|
|
return;
|
|
|
|
|
2003-11-02 10:56:48 +00:00
|
|
|
nclass = PRI_BASE(class);
|
|
|
|
oclass = PRI_BASE(kg->kg_pri_class);
|
2004-09-05 02:09:54 +00:00
|
|
|
FOREACH_THREAD_IN_GROUP(kg, td) {
|
|
|
|
ke = td->td_kse;
|
2003-04-11 03:47:14 +00:00
|
|
|
if (ke->ke_state != KES_ONRUNQ &&
|
|
|
|
ke->ke_state != KES_THREAD)
|
|
|
|
continue;
|
|
|
|
kseq = KSEQ_CPU(ke->ke_cpu);
|
|
|
|
|
2003-11-02 10:56:48 +00:00
|
|
|
#ifdef SMP
|
2003-11-15 07:32:07 +00:00
|
|
|
/*
|
|
|
|
* On SMP if we're on the RUNQ we must adjust the transferable
|
|
|
|
* count because could be changing to or from an interrupt
|
|
|
|
* class.
|
|
|
|
*/
|
|
|
|
if (ke->ke_state == KES_ONRUNQ) {
|
2004-12-26 22:56:08 +00:00
|
|
|
if (KSE_CAN_MIGRATE(ke)) {
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq->ksq_transferable--;
|
|
|
|
kseq->ksq_group->ksg_transferable--;
|
|
|
|
}
|
2004-12-26 22:56:08 +00:00
|
|
|
if (KSE_CAN_MIGRATE(ke)) {
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq->ksq_transferable++;
|
|
|
|
kseq->ksq_group->ksg_transferable++;
|
|
|
|
}
|
2003-11-15 07:32:07 +00:00
|
|
|
}
|
2003-11-02 10:56:48 +00:00
|
|
|
#endif
|
2003-11-15 07:32:07 +00:00
|
|
|
if (oclass == PRI_TIMESHARE) {
|
2003-11-02 10:56:48 +00:00
|
|
|
kseq->ksq_load_timeshare--;
|
2004-06-16 00:26:31 +00:00
|
|
|
kseq_nice_rem(kseq, kg->kg_proc->p_nice);
|
2003-11-15 07:32:07 +00:00
|
|
|
}
|
|
|
|
if (nclass == PRI_TIMESHARE) {
|
|
|
|
kseq->ksq_load_timeshare++;
|
2004-06-16 00:26:31 +00:00
|
|
|
kseq_nice_add(kseq, kg->kg_proc->p_nice);
|
2003-11-15 07:32:07 +00:00
|
|
|
}
|
2003-01-28 09:28:20 +00:00
|
|
|
}
|
|
|
|
|
2003-04-11 03:47:14 +00:00
|
|
|
kg->kg_pri_class = class;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return some of the child's priority and interactivity to the parent.
|
|
|
|
*/
|
|
|
|
void
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_exit(struct proc *p, struct thread *childtd)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_exit_ksegrp(FIRST_KSEGRP_IN_PROC(p), childtd);
|
2004-12-26 00:15:33 +00:00
|
|
|
sched_exit_thread(NULL, childtd);
|
2003-04-11 19:24:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2004-07-18 23:36:13 +00:00
|
|
|
sched_exit_ksegrp(struct ksegrp *kg, struct thread *td)
|
2003-04-11 19:24:00 +00:00
|
|
|
{
|
2004-07-18 23:36:13 +00:00
|
|
|
/* kg->kg_slptime += td->td_ksegrp->kg_slptime; */
|
|
|
|
kg->kg_runtime += td->td_ksegrp->kg_runtime;
|
2003-06-17 06:39:51 +00:00
|
|
|
sched_interact_update(kg);
|
2003-04-11 19:24:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2004-09-05 02:09:54 +00:00
|
|
|
sched_exit_thread(struct thread *td, struct thread *childtd)
|
2003-04-11 19:24:00 +00:00
|
|
|
{
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR3(KTR_SCHED, "sched_exit_thread: %p(%s) prio %d",
|
|
|
|
childtd, childtd->td_proc->p_comm, childtd->td_priority);
|
2004-09-05 02:09:54 +00:00
|
|
|
kseq_load_rem(KSEQ_CPU(childtd->td_kse->ke_cpu), childtd->td_kse);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2003-10-16 08:39:15 +00:00
|
|
|
sched_clock(struct thread *td)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
|
|
|
struct kseq *kseq;
|
2003-01-29 07:00:51 +00:00
|
|
|
struct ksegrp *kg;
|
2003-10-16 08:39:15 +00:00
|
|
|
struct kse *ke;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-06-02 05:46:48 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2004-08-10 07:52:21 +00:00
|
|
|
kseq = KSEQ_SELF();
|
2004-06-02 05:46:48 +00:00
|
|
|
#ifdef SMP
|
2004-12-26 22:56:08 +00:00
|
|
|
if (ticks >= bal_tick)
|
2004-06-02 05:46:48 +00:00
|
|
|
sched_balance();
|
2004-12-26 22:56:08 +00:00
|
|
|
if (ticks >= gbal_tick && balance_groups)
|
2004-06-02 05:46:48 +00:00
|
|
|
sched_balance_groups();
|
2004-08-10 07:52:21 +00:00
|
|
|
/*
|
|
|
|
* We could have been assigned a non real-time thread without an
|
|
|
|
* IPI.
|
|
|
|
*/
|
|
|
|
if (kseq->ksq_assigned)
|
|
|
|
kseq_assign(kseq); /* Potentially sets NEEDRESCHED */
|
2004-06-02 05:46:48 +00:00
|
|
|
#endif
|
2003-04-11 03:47:14 +00:00
|
|
|
/*
|
|
|
|
* sched_setup() apparently happens prior to stathz being set. We
|
|
|
|
* need to resolve the timers earlier in the boot so we can avoid
|
|
|
|
* calculating this here.
|
|
|
|
*/
|
|
|
|
if (realstathz == 0) {
|
|
|
|
realstathz = stathz ? stathz : hz;
|
|
|
|
tickincr = hz / realstathz;
|
|
|
|
/*
|
|
|
|
* XXX This does not work for values of stathz that are much
|
|
|
|
* larger than hz.
|
|
|
|
*/
|
|
|
|
if (tickincr == 0)
|
|
|
|
tickincr = 1;
|
|
|
|
}
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-10-16 08:39:15 +00:00
|
|
|
ke = td->td_kse;
|
2003-04-11 03:47:14 +00:00
|
|
|
kg = ke->ke_ksegrp;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-01-29 07:00:51 +00:00
|
|
|
/* Adjust ticks for pctcpu */
|
2003-03-03 05:29:09 +00:00
|
|
|
ke->ke_ticks++;
|
2003-01-28 09:30:17 +00:00
|
|
|
ke->ke_ltick = ticks;
|
2003-04-03 00:29:28 +00:00
|
|
|
|
2003-01-28 09:30:17 +00:00
|
|
|
/* Go up to one second beyond our max and then trim back down */
|
|
|
|
if (ke->ke_ftick + SCHED_CPU_TICKS + hz < ke->ke_ltick)
|
|
|
|
sched_pctcpu_update(ke);
|
|
|
|
|
2003-05-02 06:18:55 +00:00
|
|
|
if (td->td_flags & TDF_IDLETD)
|
2003-01-26 05:23:15 +00:00
|
|
|
return;
|
2003-04-11 03:47:14 +00:00
|
|
|
/*
|
2003-10-27 06:47:05 +00:00
|
|
|
* We only do slicing code for TIMESHARE ksegrps.
|
2003-04-11 03:47:14 +00:00
|
|
|
*/
|
2003-10-27 06:47:05 +00:00
|
|
|
if (kg->kg_pri_class != PRI_TIMESHARE)
|
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
2003-02-10 14:03:45 +00:00
|
|
|
* We used a tick charge it to the ksegrp so that we can compute our
|
2003-04-11 03:47:14 +00:00
|
|
|
* interactivity.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-04-11 03:47:14 +00:00
|
|
|
kg->kg_runtime += tickincr << 10;
|
2003-06-17 06:39:51 +00:00
|
|
|
sched_interact_update(kg);
|
2003-02-10 14:03:45 +00:00
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
|
|
|
* We used up one time slice.
|
|
|
|
*/
|
2003-11-17 08:24:14 +00:00
|
|
|
if (--ke->ke_slice > 0)
|
2003-04-11 03:47:14 +00:00
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
/*
|
2003-04-11 03:47:14 +00:00
|
|
|
* We're out of time, recompute priorities and requeue.
|
2003-01-26 05:23:15 +00:00
|
|
|
*/
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_rem(kseq, ke);
|
2003-04-11 03:47:14 +00:00
|
|
|
sched_priority(kg);
|
|
|
|
sched_slice(ke);
|
|
|
|
if (SCHED_CURR(kg, ke))
|
|
|
|
ke->ke_runq = kseq->ksq_curr;
|
|
|
|
else
|
|
|
|
ke->ke_runq = kseq->ksq_next;
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_add(kseq, ke);
|
2003-04-11 03:47:14 +00:00
|
|
|
td->td_flags |= TDF_NEEDRESCHED;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
sched_runnable(void)
|
|
|
|
{
|
|
|
|
struct kseq *kseq;
|
2003-06-08 00:47:33 +00:00
|
|
|
int load;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-06-08 00:47:33 +00:00
|
|
|
load = 1;
|
|
|
|
|
2003-01-29 07:00:51 +00:00
|
|
|
kseq = KSEQ_SELF();
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#ifdef SMP
|
2003-11-05 05:30:12 +00:00
|
|
|
if (kseq->ksq_assigned) {
|
|
|
|
mtx_lock_spin(&sched_lock);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
kseq_assign(kseq);
|
2003-11-05 05:30:12 +00:00
|
|
|
mtx_unlock_spin(&sched_lock);
|
|
|
|
}
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#endif
|
2003-10-27 06:47:05 +00:00
|
|
|
if ((curthread->td_flags & TDF_IDLETD) != 0) {
|
|
|
|
if (kseq->ksq_load > 0)
|
|
|
|
goto out;
|
|
|
|
} else
|
|
|
|
if (kseq->ksq_load - 1 > 0)
|
|
|
|
goto out;
|
2003-06-08 00:47:33 +00:00
|
|
|
load = 0;
|
|
|
|
out:
|
|
|
|
return (load);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
sched_userret(struct thread *td)
|
|
|
|
{
|
|
|
|
struct ksegrp *kg;
|
|
|
|
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
KASSERT((td->td_flags & TDF_BORROWING) == 0,
|
|
|
|
("thread with borrowed priority returning to userland"));
|
|
|
|
kg = td->td_ksegrp;
|
|
|
|
if (td->td_priority != kg->kg_user_pri) {
|
2003-01-26 05:23:15 +00:00
|
|
|
mtx_lock_spin(&sched_lock);
|
|
|
|
td->td_priority = kg->kg_user_pri;
|
Rework the interface between priority propagation (lending) and the
schedulers a bit to ensure more correct handling of priorities and fewer
priority inversions:
- Add two functions to the sched(9) API to handle priority lending:
sched_lend_prio() and sched_unlend_prio(). The turnstile code uses these
functions to ask the scheduler to lend a thread a set priority and to
tell the scheduler when it thinks it is ok for a thread to stop borrowing
priority. The unlend case is slightly complex in that the turnstile code
tells the scheduler what the minimum priority of the thread needs to be
to satisfy the requirements of any other threads blocked on locks owned
by the thread in question. The scheduler then decides where the thread
can go back to normal mode (if it's normal priority is high enough to
satisfy the pending lock requests) or it it should continue to use the
priority specified to the sched_unlend_prio() call. This involves adding
a new per-thread flag TDF_BORROWING that replaces the ULE-only kse flag
for priority elevation.
- Schedulers now refuse to lower the priority of a thread that is currently
borrowing another therad's priority.
- If a scheduler changes the priority of a thread that is currently sitting
on a turnstile, it will call a new function turnstile_adjust() to inform
the turnstile code of the change. This function resorts the thread on
the priority list of the turnstile if needed, and if the thread ends up
at the head of the list (due to having the highest priority) and its
priority was raised, then it will propagate that new priority to the
owner of the lock it is blocked on.
Some additional fixes specific to the 4BSD scheduler include:
- Common code for updating the priority of a thread when the user priority
of its associated kse group has been consolidated in a new static
function resetpriority_thread(). One change to this function is that
it will now only adjust the priority of a thread if it already has a
time sharing priority, thus preserving any boosts from a tsleep() until
the thread returns to userland. Also, resetpriority() no longer calls
maybe_resched() on each thread in the group. Instead, the code calling
resetpriority() is responsible for calling resetpriority_thread() on
any threads that need to be updated.
- schedcpu() now uses resetpriority_thread() instead of just calling
sched_prio() directly after it updates a kse group's user priority.
- sched_clock() now uses resetpriority_thread() rather than writing
directly to td_priority.
- sched_nice() now updates all the priorities of the threads after the
group priority has been adjusted.
Discussed with: bde
Reviewed by: ups, jeffr
Tested on: 4bsd, ule
Tested on: i386, alpha, sparc64
2004-12-30 20:52:44 +00:00
|
|
|
td->td_base_pri = kg->kg_user_pri;
|
2003-01-26 05:23:15 +00:00
|
|
|
mtx_unlock_spin(&sched_lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2003-01-28 09:28:20 +00:00
|
|
|
struct kse *
|
|
|
|
sched_choose(void)
|
|
|
|
{
|
2003-01-29 07:00:51 +00:00
|
|
|
struct kseq *kseq;
|
2003-01-28 09:28:20 +00:00
|
|
|
struct kse *ke;
|
|
|
|
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
kseq = KSEQ_SELF();
|
2003-04-11 03:47:14 +00:00
|
|
|
#ifdef SMP
|
2003-12-11 03:57:10 +00:00
|
|
|
restart:
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
if (kseq->ksq_assigned)
|
|
|
|
kseq_assign(kseq);
|
2003-04-11 03:47:14 +00:00
|
|
|
#endif
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
ke = kseq_choose(kseq);
|
2003-01-26 05:23:15 +00:00
|
|
|
if (ke) {
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#ifdef SMP
|
|
|
|
if (ke->ke_ksegrp->kg_pri_class == PRI_IDLE)
|
2003-12-11 03:57:10 +00:00
|
|
|
if (kseq_idled(kseq) == 0)
|
|
|
|
goto restart;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#endif
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_runq_rem(kseq, ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
ke->ke_state = KES_THREAD;
|
2003-04-11 03:47:14 +00:00
|
|
|
return (ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
2003-01-28 09:28:20 +00:00
|
|
|
#ifdef SMP
|
2003-12-11 03:57:10 +00:00
|
|
|
if (kseq_idled(kseq) == 0)
|
|
|
|
goto restart;
|
2003-01-28 09:28:20 +00:00
|
|
|
#endif
|
2003-04-11 03:47:14 +00:00
|
|
|
return (NULL);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2004-09-01 02:11:28 +00:00
|
|
|
sched_add(struct thread *td, int flags)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2003-02-03 05:30:07 +00:00
|
|
|
struct kseq *kseq;
|
2003-04-11 03:47:14 +00:00
|
|
|
struct ksegrp *kg;
|
2003-10-16 08:39:15 +00:00
|
|
|
struct kse *ke;
|
2004-12-26 22:56:08 +00:00
|
|
|
int preemptive;
|
2004-08-10 07:52:21 +00:00
|
|
|
int canmigrate;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
int class;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR5(KTR_SCHED, "sched_add: %p(%s) prio %d by %p(%s)",
|
|
|
|
td, td->td_proc->p_comm, td->td_priority, curthread,
|
|
|
|
curthread->td_proc->p_comm);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
2003-10-16 08:39:15 +00:00
|
|
|
ke = td->td_kse;
|
|
|
|
kg = td->td_ksegrp;
|
2004-12-26 22:56:08 +00:00
|
|
|
canmigrate = 1;
|
|
|
|
preemptive = !(flags & SRQ_YIELDING);
|
|
|
|
class = PRI_BASE(kg->kg_pri_class);
|
|
|
|
kseq = KSEQ_SELF();
|
|
|
|
if ((ke->ke_flags & KEF_INTERNAL) == 0)
|
|
|
|
SLOT_USE(td->td_ksegrp);
|
|
|
|
ke->ke_flags &= ~KEF_INTERNAL;
|
|
|
|
#ifdef SMP
|
2004-12-13 13:09:33 +00:00
|
|
|
if (ke->ke_flags & KEF_ASSIGNED) {
|
2004-12-26 22:56:08 +00:00
|
|
|
if (ke->ke_flags & KEF_REMOVED)
|
2004-12-13 13:09:33 +00:00
|
|
|
ke->ke_flags &= ~KEF_REMOVED;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
return;
|
2004-12-13 13:09:33 +00:00
|
|
|
}
|
2004-12-26 22:56:08 +00:00
|
|
|
canmigrate = KSE_CAN_MIGRATE(ke);
|
|
|
|
#endif
|
2003-01-26 05:23:15 +00:00
|
|
|
KASSERT(ke->ke_state != KES_ONRUNQ,
|
2003-02-03 05:30:07 +00:00
|
|
|
("sched_add: kse %p (%s) already in run queue", ke,
|
2003-01-26 05:23:15 +00:00
|
|
|
ke->ke_proc->p_comm));
|
|
|
|
KASSERT(ke->ke_proc->p_sflag & PS_INMEM,
|
2003-02-03 05:30:07 +00:00
|
|
|
("sched_add: process swapped out"));
|
2003-04-12 07:28:36 +00:00
|
|
|
KASSERT(ke->ke_runq == NULL,
|
|
|
|
("sched_add: KSE %p is still assigned to a run queue", ke));
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
switch (class) {
|
2003-04-03 00:29:28 +00:00
|
|
|
case PRI_ITHD:
|
|
|
|
case PRI_REALTIME:
|
2003-04-11 03:47:14 +00:00
|
|
|
ke->ke_runq = kseq->ksq_curr;
|
|
|
|
ke->ke_slice = SCHED_SLICE_MAX;
|
2004-12-26 22:56:08 +00:00
|
|
|
if (canmigrate)
|
|
|
|
ke->ke_cpu = PCPU_GET(cpuid);
|
2003-04-03 00:29:28 +00:00
|
|
|
break;
|
|
|
|
case PRI_TIMESHARE:
|
2003-04-12 07:28:36 +00:00
|
|
|
if (SCHED_CURR(kg, ke))
|
|
|
|
ke->ke_runq = kseq->ksq_curr;
|
|
|
|
else
|
|
|
|
ke->ke_runq = kseq->ksq_next;
|
2003-04-11 03:47:14 +00:00
|
|
|
break;
|
2003-04-03 00:29:28 +00:00
|
|
|
case PRI_IDLE:
|
2003-04-11 03:47:14 +00:00
|
|
|
/*
|
|
|
|
* This is for priority prop.
|
|
|
|
*/
|
2003-10-27 06:47:05 +00:00
|
|
|
if (ke->ke_thread->td_priority < PRI_MIN_IDLE)
|
2003-04-11 03:47:14 +00:00
|
|
|
ke->ke_runq = kseq->ksq_curr;
|
|
|
|
else
|
|
|
|
ke->ke_runq = &kseq->ksq_idle;
|
|
|
|
ke->ke_slice = SCHED_SLICE_MIN;
|
|
|
|
break;
|
|
|
|
default:
|
2003-11-02 03:36:33 +00:00
|
|
|
panic("Unknown pri class.");
|
2003-04-03 00:29:28 +00:00
|
|
|
break;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#ifdef SMP
|
2004-08-10 07:52:21 +00:00
|
|
|
/*
|
|
|
|
* Don't migrate running threads here. Force the long term balancer
|
|
|
|
* to do it.
|
|
|
|
*/
|
2004-08-12 07:56:33 +00:00
|
|
|
if (ke->ke_flags & KEF_HOLD) {
|
|
|
|
ke->ke_flags &= ~KEF_HOLD;
|
2004-08-10 07:52:21 +00:00
|
|
|
canmigrate = 0;
|
2004-08-12 07:56:33 +00:00
|
|
|
}
|
2004-08-10 07:52:21 +00:00
|
|
|
/*
|
|
|
|
* If this thread is pinned or bound, notify the target cpu.
|
|
|
|
*/
|
|
|
|
if (!canmigrate && ke->ke_cpu != PCPU_GET(cpuid) ) {
|
2003-12-14 02:06:29 +00:00
|
|
|
ke->ke_runq = NULL;
|
2003-12-11 03:57:10 +00:00
|
|
|
kseq_notify(ke, ke->ke_cpu);
|
|
|
|
return;
|
|
|
|
}
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
/*
|
2003-12-20 14:03:14 +00:00
|
|
|
* If we had been idle, clear our bit in the group and potentially
|
|
|
|
* the global bitmap. If not, see if we should transfer this thread.
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
*/
|
2003-12-11 03:57:10 +00:00
|
|
|
if ((class == PRI_TIMESHARE || class == PRI_REALTIME) &&
|
|
|
|
(kseq->ksq_group->ksg_idlemask & PCPU_GET(cpumask)) != 0) {
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
/*
|
2003-12-11 03:57:10 +00:00
|
|
|
* Check to see if our group is unidling, and if so, remove it
|
|
|
|
* from the global idle mask.
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
*/
|
2003-12-11 03:57:10 +00:00
|
|
|
if (kseq->ksq_group->ksg_idlemask ==
|
|
|
|
kseq->ksq_group->ksg_cpumask)
|
|
|
|
atomic_clear_int(&kseq_idle, kseq->ksq_group->ksg_mask);
|
|
|
|
/*
|
|
|
|
* Now remove ourselves from the group specific idle mask.
|
|
|
|
*/
|
|
|
|
kseq->ksq_group->ksg_idlemask &= ~PCPU_GET(cpumask);
|
2004-12-26 22:56:08 +00:00
|
|
|
} else if (canmigrate && kseq->ksq_load > 1 && class != PRI_ITHD)
|
2003-12-20 14:03:14 +00:00
|
|
|
if (kseq_transfer(kseq, ke, class))
|
|
|
|
return;
|
2004-08-10 07:52:21 +00:00
|
|
|
ke->ke_cpu = PCPU_GET(cpuid);
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
#endif
|
2004-08-12 07:56:33 +00:00
|
|
|
if (td->td_priority < curthread->td_priority &&
|
|
|
|
ke->ke_runq == kseq->ksq_curr)
|
|
|
|
curthread->td_flags |= TDF_NEEDRESCHED;
|
2004-07-08 21:45:04 +00:00
|
|
|
if (preemptive && maybe_preempt(td))
|
2004-07-02 20:21:44 +00:00
|
|
|
return;
|
2003-01-26 05:23:15 +00:00
|
|
|
ke->ke_state = KES_ONRUNQ;
|
|
|
|
|
2004-12-26 22:56:08 +00:00
|
|
|
kseq_runq_add(kseq, ke, flags);
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_add(kseq, ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
2003-10-16 08:39:15 +00:00
|
|
|
sched_rem(struct thread *td)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
2003-04-11 03:47:14 +00:00
|
|
|
struct kseq *kseq;
|
2003-10-16 08:39:15 +00:00
|
|
|
struct kse *ke;
|
|
|
|
|
2004-12-26 00:15:33 +00:00
|
|
|
CTR5(KTR_SCHED, "sched_rem: %p(%s) prio %d by %p(%s)",
|
|
|
|
td, td->td_proc->p_comm, td->td_priority, curthread,
|
|
|
|
curthread->td_proc->p_comm);
|
2004-12-26 22:56:08 +00:00
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
ke = td->td_kse;
|
|
|
|
SLOT_RELEASE(td->td_ksegrp);
|
2004-12-13 13:09:33 +00:00
|
|
|
if (ke->ke_flags & KEF_ASSIGNED) {
|
|
|
|
ke->ke_flags |= KEF_REMOVED;
|
- Add static to local functions and data where it was missing.
- Add an IPI based mechanism for migrating kses. This mechanism is
broken down into several components. This is intended to reduce cache
thrashing by eliminating most cases where one cpu touches another's
run queues.
- kseq_notify() appends a kse to a lockless singly linked list and
conditionally sends an IPI to the target processor. Right now this is
protected by sched_lock but at some point I'd like to get rid of the
global lock. This is why I used something more complicated than a
standard queue.
- kseq_assign() processes our list of kses that have been assigned to us
by other processors. This simply calls sched_add() for each item on the
list after clearing the new KEF_ASSIGNED flag. This flag is used to
indicate that we have been appeneded to the assigned queue but not
added to the run queue yet.
- In sched_add(), instead of adding a KSE to another processor's queue we
use kse_notify() so that we don't touch their queue. Also in sched_add(),
if KEF_ASSIGNED is already set return immediately. This can happen if
a thread is removed and readded so that the priority is recorded properly.
- In sched_rem() return immediately if KEF_ASSIGNED is set. All callers
immediately readd simply to adjust priorites etc.
- In sched_choose(), if we're running an IDLE task or the per cpu idle thread
set our cpumask bit in 'kseq_idle' so that other processors may know that
we are idle. Before this, make a single pass through the run queues of
other processors so that we may find work more immediately if it is
available.
- In sched_runnable(), don't scan each processor's run queue, they will IPI
us if they have work for us to do.
- In sched_add(), if we're adding a thread that can be migrated and we have
plenty of work to do, try to migrate the thread to an idle kseq.
- Simplify the logic in sched_prio() and take the KEF_ASSIGNED flag into
consideration.
- No longer use kseq_choose() to steal threads, it can lose it's last
argument.
- Create a new function runq_steal() which operates like runq_choose() but
skips threads based on some criteria. Currently it will not steal
PRI_ITHD threads. In the future this will be used for CPU binding.
- Create a kseq_steal() that checks each run queue with runq_steal(), use
kseq_steal() in the places where we used kseq_choose() to steal with
before.
2003-10-31 11:16:04 +00:00
|
|
|
return;
|
2004-12-13 13:09:33 +00:00
|
|
|
}
|
2004-01-25 08:57:38 +00:00
|
|
|
KASSERT((ke->ke_state == KES_ONRUNQ),
|
|
|
|
("sched_rem: KSE not on run queue"));
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2004-12-13 13:09:33 +00:00
|
|
|
ke->ke_state = KES_THREAD;
|
2003-04-11 03:47:14 +00:00
|
|
|
kseq = KSEQ_CPU(ke->ke_cpu);
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_runq_rem(kseq, ke);
|
|
|
|
kseq_load_rem(kseq, ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
fixpt_t
|
2003-10-16 08:39:15 +00:00
|
|
|
sched_pctcpu(struct thread *td)
|
2003-01-26 05:23:15 +00:00
|
|
|
{
|
|
|
|
fixpt_t pctcpu;
|
2003-10-16 08:39:15 +00:00
|
|
|
struct kse *ke;
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
pctcpu = 0;
|
2003-10-16 08:39:15 +00:00
|
|
|
ke = td->td_kse;
|
2003-10-20 19:55:21 +00:00
|
|
|
if (ke == NULL)
|
|
|
|
return (0);
|
2003-01-26 05:23:15 +00:00
|
|
|
|
2003-06-08 00:47:33 +00:00
|
|
|
mtx_lock_spin(&sched_lock);
|
2003-01-26 05:23:15 +00:00
|
|
|
if (ke->ke_ticks) {
|
|
|
|
int rtick;
|
|
|
|
|
2003-06-15 02:18:29 +00:00
|
|
|
/*
|
|
|
|
* Don't update more frequently than twice a second. Allowing
|
|
|
|
* this causes the cpu usage to decay away too quickly due to
|
|
|
|
* rounding errors.
|
|
|
|
*/
|
2003-12-11 04:23:39 +00:00
|
|
|
if (ke->ke_ftick + SCHED_CPU_TICKS < ke->ke_ltick ||
|
|
|
|
ke->ke_ltick < (ticks - (hz / 2)))
|
2003-06-15 02:18:29 +00:00
|
|
|
sched_pctcpu_update(ke);
|
2003-01-26 05:23:15 +00:00
|
|
|
/* How many rtick per second ? */
|
2003-06-15 02:18:29 +00:00
|
|
|
rtick = min(ke->ke_ticks / SCHED_CPU_TIME, SCHED_CPU_TICKS);
|
2003-02-02 08:24:32 +00:00
|
|
|
pctcpu = (FSCALE * ((FSCALE * rtick)/realstathz)) >> FSHIFT;
|
2003-01-26 05:23:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
ke->ke_proc->p_swtime = ke->ke_ltick - ke->ke_ftick;
|
2003-04-22 19:48:25 +00:00
|
|
|
mtx_unlock_spin(&sched_lock);
|
2003-01-26 05:23:15 +00:00
|
|
|
|
|
|
|
return (pctcpu);
|
|
|
|
}
|
|
|
|
|
2003-11-04 07:45:41 +00:00
|
|
|
void
|
|
|
|
sched_bind(struct thread *td, int cpu)
|
|
|
|
{
|
|
|
|
struct kse *ke;
|
|
|
|
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
ke = td->td_kse;
|
|
|
|
ke->ke_flags |= KEF_BOUND;
|
2003-12-11 03:57:10 +00:00
|
|
|
#ifdef SMP
|
|
|
|
if (PCPU_GET(cpuid) == cpu)
|
2003-11-04 07:45:41 +00:00
|
|
|
return;
|
|
|
|
/* sched_rem without the runq_remove */
|
|
|
|
ke->ke_state = KES_THREAD;
|
2003-11-15 07:32:07 +00:00
|
|
|
kseq_load_rem(KSEQ_CPU(ke->ke_cpu), ke);
|
2003-11-04 07:45:41 +00:00
|
|
|
kseq_notify(ke, cpu);
|
|
|
|
/* When we return from mi_switch we'll be on the correct cpu. */
|
2004-07-03 16:57:51 +00:00
|
|
|
mi_switch(SW_VOL, NULL);
|
2003-11-04 07:45:41 +00:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
sched_unbind(struct thread *td)
|
|
|
|
{
|
|
|
|
mtx_assert(&sched_lock, MA_OWNED);
|
|
|
|
td->td_kse->ke_flags &= ~KEF_BOUND;
|
|
|
|
}
|
|
|
|
|
2004-02-01 02:48:36 +00:00
|
|
|
int
|
|
|
|
sched_load(void)
|
|
|
|
{
|
|
|
|
#ifdef SMP
|
|
|
|
int total;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
total = 0;
|
|
|
|
for (i = 0; i <= ksg_maxid; i++)
|
|
|
|
total += KSEQ_GROUP(i)->ksg_load;
|
|
|
|
return (total);
|
|
|
|
#else
|
|
|
|
return (KSEQ_SELF()->ksq_sysload);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2003-01-26 05:23:15 +00:00
|
|
|
int
|
|
|
|
sched_sizeof_ksegrp(void)
|
|
|
|
{
|
|
|
|
return (sizeof(struct ksegrp) + sizeof(struct kg_sched));
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
sched_sizeof_proc(void)
|
|
|
|
{
|
|
|
|
return (sizeof(struct proc));
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
sched_sizeof_thread(void)
|
|
|
|
{
|
|
|
|
return (sizeof(struct thread) + sizeof(struct td_sched));
|
|
|
|
}
|
2004-09-05 02:09:54 +00:00
|
|
|
#define KERN_SWITCH_INCLUDE 1
|
|
|
|
#include "kern/kern_switch.c"
|