REAL-TIME（SCHED_FIFO和SCHED_RR）进程会导致系统LOCKUP吗？
目录
Linux kernel支持两种实时(real-time)调度策略(scheduling policy)：SCHED_FIFO和SCHED_RR，无论是哪一种，实时进程的优先级范围[0~99]都高于普通进程[100~139]，始终优先于普通进程得到运行。如果实时进程是CPU消耗型的，会不会导致其它进程得不到运行机会，造成系统lockup呢？
这实际上是两个问题，不能混为一谈，第一个问题是会不会造成系统lockup，第二个问题是会不会导致其它进程得不到运行机会。我们一个一个分别来谈。
实时进程会不会造成系统lockup？可能会
Lockup分为soft lockup和hard lockup，我在《内核如何检测SOFT LOCKUP与HARD LOCKUP》一文中解释了Linux kernel检测lockup的原理。
Hard lockup发生在CPU中断被屏蔽的情况下，因为实时进程本身并不会屏蔽CPU中断，hrtimer时钟中断是可以得到响应的，所以不会导致hard lockup。
Soft lockup发生在内核线程[watchdog/x]得不到运行的情况下，理论上如果实时进程占着CPU不放，确实有可能导致[watchdog/x]得不到运行而发生soft lockup，然而这个可能性并不大，因为[watchdog/x]本身也是实时进程，调度策略为SCHED_FIFO，优先级已经是最高的99：
$ ps -ef | grep watchdog
root 6 2 0 Feb24 ? 00:00:18 [watchdog/0]
root 10 2 0 Feb24 ? 00:00:16 [watchdog/1]
root 14 2 0 Feb24 ? 00:00:13 [watchdog/2]
root 18 2 0 Feb24 ? 00:00:12 [watchdog/3]
$ chrt -p 6
pid 6's current scheduling policy: SCHED_FIFO
pid 6's current scheduling priority: 99如果占着CPU不放的实时进程也是SCHED_FIFO并且优先级为99，就有可能导致soft lockup。为什么呢？我们看一下实时进程的调度策略就明白了：
- 在多个实时进程之间，优先级更高的会抢先运行（注：实时进程的优先级数字越大则优先级越高，99最高，0最低；而普通进程正好相反，优先级数字越大则优先级越低，139最低，100最高）；
- 优先级相同的实时进程之间，不会互相抢占，只能等对方主动释放CPU；
- SCHED_FIFO调度策略的特点是，进程会一直保持运行直到发生以下情况之一：
- 进程主动调用sched_yield(2)放弃运行，自动排到运行队列的队尾，等到相同优先级的其它进程运行之后才有机会再运行；
- 进程进入睡眠状态（比如由于等待I/O的原因），唤醒后自动排到运行队列的队尾，等到相同优先级的其它进程运行之后才有机会再运行；
- 被优先级更高的实时进程抢占，这种情况下会自动排到运行队列的队首，下次运行的机会排在相同优先级的其它进程的前面。
- SCHED_RR进程与SCHED_FIFO唯一不同的是，实时进程的运行时间是分为一段一段的，在相同优先级的进程之间轮流运行，每个进程运行完一个时间段之后，必须让给下一个进程（强调：仅对相同优先级而言，不同优先级的进程之间仍然会互相抢占）。
所以，如果占着CPU不放的实时进程的调度策略是SCHED_FIFO，并且优先级为与[watchdog/x]相同的99，SCHED_FIFO的调度策略决定了只要它不放手，[watchdog/x]就无法运行，结果是会导致soft lockup。
接下来第二个问题是：
实时进程会不会导致其它进程得不到运行机会？不会
如果实时进程占着CPU不放，会不会导致其它进程得不到运行机会，包括管理员的shell也无法运行、连基本的管理任务也进行不了，最终造成整个系统失去控制？
通常不会。因为Linux kernel有一个RealTime Throttling机制，就是为了防止CPU消耗型的实时进程霸占所有的CPU资源而造成整个系统失去控制。它的原理很简单，就是保证无论如何普通进程都能得到一定比例（默认5%）的CPU时间，可以通过两个内核参数来控制：
- /proc/sys/kernel/sched_rt_period_us
缺省值是1,000,000 μs (1秒)，表示实时进程的运行粒度为1秒。（注：修改这个参数请谨慎，太大或太小都可能带来问题）。 - /proc/sys/kernel/sched_rt_runtime_us
缺省值是 950,000 μs (0.95秒)，表示在1秒的运行周期里所有的实时进程一起最多可以占用0.95秒的CPU时间。
如果sched_rt_runtime_us=-1，表示取消限制，意味着实时进程可以占用100%的CPU时间（慎用，有可能使系统失去控制）。
所以，Linux kernel默认情况下保证了普通进程无论如何都可以得到5%的CPU时间，尽管系统可能会慢如蜗牛，但管理员仍然可以利用这5%的时间设法恢复系统，比如停掉失控的实时进程，或者给自己的shell进程赋予更高的实时优先级以便执行管理任务，等等。
Real-time Throttling支持cgroup，详见https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt
SCHED_FIFO和实时节流
https://lwn.net/Articles/296419/
SCHED_FIFO调度类是一个长期的，由POSIX指定的实时功能。在此类进程中，只要有需要，便为它们分配CPU的时间，这仅取决于更高优先级的实时进程的需求。如果有两个具有相同优先级的SCHED_FIFO进程争用CPU，则当前正在运行的进程将继续这样做，直到决定放弃处理器为止。SCHED_FIFO因此对于实时应用非常有用，在实时应用中，人们要非常有把握地知道，系统上最高优先级的进程将在需要时可以完全访问处理器。
在2.6.25周期中合并的众多功能之一是实时组调度。为了平衡竞争的进程组之间的CPU使用率，每个进程组都可以运行实时任务，组调度程序引入了“实时带宽”或rt_bandwith的概念。此带宽由一对值组成：CPU时间计费周期，以及该时间段内允许组使用的CPU数量（以实时优先级）。一旦SCHED_FIFO任务导致某个组超过其rt_bandwidth，无论是否希望将其从处理器中推出。
如果要允许多个组拆分系统的实时处理能力，则需要此功能。但事实证明，它也可以用在默认情况下，即系统上的所有进程都包含在一个默认组中。自2.6.25开始发布的内核已将默认组的rt_bandwidth值设置为每1.0秒中的0.95。换句话说，默认情况下，将组调度程序配置为为非SCHED_FIFO任务保留5％的CPU。
直到8月中旬，Peter Zijlstra发布了一个将默认值设置为“ unlimited”的补丁程序之后，似乎没有人真正注意到此功能。到那时，很明显，一些开发人员对如何设置这种策略的想法与其他开发者不同。
Ingo Molnar不赞成该补丁，他说：
Ingo的建议是 将限制提高到十秒的CPU时间。正如他（和其他人）指出的那样：任何需要长时间独占CPU的SCHED_FIFO应用程序都存在严重的问题，需要修复。
让SCHED_FIFO进程无限期地运行确实存在一些问题。如果该进程永远不会放弃CPU，那么系统将永远挂死；管理员不可能使用kill命令。这个过程还将阻止重要的事情，例如内核线程。即使十秒钟后释放处理器，它也会严重降低系统其余部分的运行。即使在多处理器系统上，通常也会有运行SCHED_FIFO进程的绑定到CPU的进程。在不破坏其CPU亲和力的情况下将无法恢复这些进程，这不是任何人都想采取的步骤。
因此，有人认为rt_bandwidth限制是一个重要的安全破坏者。有了它，即使是失控的SCHED_FIFO也无法阻止管理员（最终）重新获得对系统的控制并弄清楚正在发生的事情。作为安全性的交换，此功能仅占用少量CPU时间的SCHED_FIFO任务-等同于在稍弱的处理器上运行应用程序。
那些反对默认rt_bandwidth限制的人指出了两个要点：这是用户空间API的更改（这也破坏了POSIX的合规性），并表示内核对策略的强加。首先，尼克·皮金（Nick Piggin）担心此更改可能导致应用程序损坏：
或实时应用程序肯定可以自适应地使用CPU的100％，但仍然无法忍受意外的抢占。
可能使问题更严重的是，在测试过程中节气门可能未切入。相反，它可以等到生产系统中出现意外情况。不用说，对于创建和部署这种系统的人来说，这是一个可怕的前景。
Linus拒绝了 “内核中的策略”参数，他指出内核中有很多策略，尤其是在可调参数的默认设置方面。他说：
Linus小心翼翼地避免采取对大多数人来说有意义的设置立场。可以肯定地说，使系统能够抵抗失控的实时过程接管是更明智的设置，尤其是考虑到对运行具有实时优先级的令人讨厌的应用程序（如PulseAudio）有一定的兴趣。另一方面，还可以证明符合标准（和预期的）SCHED_FIFO语义是唯一有意义的选择。
有人谈论过创建一个新的实时调度类，其中节流是其语义的一部分。该类甚至可以以较低的限制提供给非特权进程。同时，在撰写本文时，0.95秒的限制（似乎没人喜欢的一个选择） 保持不变。几乎可以肯定它会被提出；我们需要等待多少钱。
参考资料
《NMI watchdog: BUG: soft lockup》
Real-Time group scheduling
https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt
Real-Time group scheduling
--------------------------
CONTENTS
========
0. WARNING
1. Overview
1.1 The problem
1.2 The solution
2. The interface
2.1 System-wide settings
2.2 Default behaviour
2.3 Basis for grouping tasks
3. Future plans
0. WARNING
==========
Fiddling with these settings can result in an unstable system, the knobs are
root only and assumes root knows what he is doing.
Most notable:
* very small values in sched_rt_period_us can result in an unstable
system when the period is smaller than either the available hrtimer
resolution, or the time it takes to handle the budget refresh itself.
* very small values in sched_rt_runtime_us can result in an unstable
system when the runtime is so small the system has difficulty making
forward progress (NOTE: the migration thread and kstopmachine both
are real-time processes).
1. Overview
===========
1.1 The problem
---------------
Realtime scheduling is all about determinism, a group has to be able to rely on
the amount of bandwidth (eg. CPU time) being constant. In order to schedule
multiple groups of realtime tasks, each group must be assigned a fixed portion
of the CPU time available. Without a minimum guarantee a realtime group can
obviously fall short. A fuzzy upper limit is of no use since it cannot be
relied upon. Which leaves us with just the single fixed portion.
1.2 The solution
----------------
CPU time is divided by means of specifying how much time can be spent running
in a given period. We allocate this "run time" for each realtime group which
the other realtime groups will not be permitted to use.
Any time not allocated to a realtime group will be used to run normal priority
tasks (SCHED_OTHER). Any allocated run time not used will also be picked up by
SCHED_OTHER.
Let's consider an example: a frame fixed realtime renderer must deliver 25
frames a second, which yields a period of 0.04s per frame. Now say it will also
have to play some music and respond to input, leaving it with around 80% CPU
time dedicated for the graphics. We can then give this group a run time of 0.8
* 0.04s = 0.032s.
This way the graphics group will have a 0.04s period with a 0.032s run time
limit. Now if the audio thread needs to refill the DMA buffer every 0.005s, but
needs only about 3% CPU time to do so, it can do with a 0.03 * 0.005s =
0.00015s. So this group can be scheduled with a period of 0.005s and a run time
of 0.00015s.
The remaining CPU time will be used for user input and other tasks. Because
realtime tasks have explicitly allocated the CPU time they need to perform
their tasks, buffer underruns in the graphics or audio can be eliminated.
NOTE: the above example is not fully implemented yet. We still
lack an EDF scheduler to make non-uniform periods usable.
2. The Interface
================
2.1 System wide settings
------------------------
The system wide settings are configured under the /proc virtual file system:
/proc/sys/kernel/sched_rt_period_us:
The scheduling period that is equivalent to 100% CPU bandwidth
/proc/sys/kernel/sched_rt_runtime_us:
A global limit on how much time realtime scheduling may use. Even without
CONFIG_RT_GROUP_SCHED enabled, this will limit time reserved to realtime
processes. With CONFIG_RT_GROUP_SCHED it signifies the total bandwidth
available to all realtime groups.
* Time is specified in us because the interface is s32. This gives an
operating range from 1us to about 35 minutes.
* sched_rt_period_us takes values from 1 to INT_MAX.
* sched_rt_runtime_us takes values from -1 to (INT_MAX - 1).
* A run time of -1 specifies runtime == period, ie. no limit.
2.2 Default behaviour
---------------------
The default values for sched_rt_period_us (1000000 or 1s) and
sched_rt_runtime_us (950000 or 0.95s). This gives 0.05s to be used by
SCHED_OTHER (non-RT tasks). These defaults were chosen so that a run-away
realtime tasks will not lock up the machine but leave a little time to recover
it. By setting runtime to -1 you'd get the old behaviour back.
By default all bandwidth is assigned to the root group and new groups get the
period from /proc/sys/kernel/sched_rt_period_us and a run time of 0. If you
want to assign bandwidth to another group, reduce the root group's bandwidth
and assign some or all of the difference to another group.
Realtime group scheduling means you have to assign a portion of total CPU
bandwidth to the group before it will accept realtime tasks. Therefore you will
not be able to run realtime tasks as any user other than root until you have
done that, even if the user has the rights to run processes with realtime
priority!
2.3 Basis for grouping tasks
----------------------------
Enabling CONFIG_RT_GROUP_SCHED lets you explicitly allocate real
CPU bandwidth to task groups.
This uses the cgroup virtual file system and "<cgroup>/cpu.rt_runtime_us"
to control the CPU time reserved for each control group.
For more information on working with control groups, you should read
Documentation/cgroup-v1/cgroups.txt as well.
Group settings are checked against the following limits in order to keep the
configuration schedulable:
\Sum_{i} runtime_{i} / global_period <= global_runtime / global_period
For now, this can be simplified to just the following (but see Future plans):
\Sum_{i} runtime_{i} <= global_runtime
3. Future plans
===============
There is work in progress to make the scheduling period for each group
("<cgroup>/cpu.rt_period_us") configurable as well.
The constraint on the period is that a subgroup must have a smaller or
equal period to its parent. But realistically its not very useful _yet_
as its prone to starvation without deadline scheduling.
Consider two sibling groups A and B; both have 50% bandwidth, but A's
period is twice the length of B's.
* group A: period=100000us, runtime=50000us
- this runs for 0.05s once every 0.1s
* group B: period= 50000us, runtime=25000us
- this runs for 0.025s twice every 0.1s (or once every 0.05 sec).
This means that currently a while (1) loop in A will run for the full period of
B and can starve B's tasks (assuming they are of lower priority) for a whole
period.
The next project will be SCHED_EDF (Earliest Deadline First scheduling) to bring
full deadline scheduling to the linux kernel. Deadline scheduling the above
groups and treating end of the period as a deadline will ensure that they both
get their allocated time.
Implementing SCHED_EDF might take a while to complete. Priority Inheritance is
the biggest challenge as the current linux PI infrastructure is geared towards
the limited static priority levels 0-99. With deadline scheduling you need to
do deadline inheritance (since priority is inversely proportional to the
deadline delta (deadline - now)).
This means the whole PI machinery will have to be reworked - and that is one of
the most complex pieces of code we have.