24  * struct rcu_tasks_percpu - Per-CPU component of definition for a Tasks-RCU-like mechanism.
34  * @rtp_blkd_tasks: List of tasks blocked as readers.
35  * @rtp_exit_list: List of tasks in the latter portion of do_exit().
58  * struct rcu_tasks - Definition for a Tasks-RCU-like mechanism.
193 /* RCU tasks grace-period state for debugging. */
250 // Tasks RCU.  Do not enqueue callbacks before this function is invoked.
344 // Enqueue a callback for the specified flavor of Tasks RCU.
420 // Wait for all in-flight callbacks for the specified RCU Tasks flavor.
631 // RCU-tasks kthread that detects grace periods and invokes callbacks.
651 	 * one RCU-tasks grace period and then invokes the callbacks.  in rcu_tasks_kthread()
664 // Wait for a grace period for the specified flavor of Tasks RCU.
680 /* Spawn RCU-tasks grace-period kthread. */
694  * Print any non-default Tasks RCU settings.
710 	pr_info("\tTrampoline variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
713 	pr_info("\tRude variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
716 	pr_info("\tTracing variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
721 /* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */
755 /* Dump out more rcutorture-relevant state common to all RCU-tasks flavors. */
764 	pr_alert("%s%s Tasks%s RCU g%ld gp_start %lu gp_jiffies %lu gp_state %d (%s).\n",  in rcu_tasks_torture_stats_print_generic()
809 // Shared code between task-list-scanning variants of Tasks RCU.
811 /* Wait for one RCU-tasks grace period. */
828 	 * There were callbacks, so we need to wait for an RCU-tasks  in rcu_tasks_wait_gp()
829 	 * grace period.  Start off by scanning the task list for tasks  in rcu_tasks_wait_gp()
830 	 * that are not already voluntarily blocked.  Mark these tasks  in rcu_tasks_wait_gp()
846 	 * tasks, removing any that are no longer holdouts.  When the list  in rcu_tasks_wait_gp()
931 //	Gather per-CPU lists of tasks in do_exit() to ensure that all
932 //	tasks that were in the process of exiting (and which thus might
933 //	not know to synchronize with this RCU Tasks grace period) have
935 //	will take care of any tasks stuck in the non-preemptible region
943 //	t->on_rq and t->nvcsw transitions are seen by all CPUs and tasks
944 //	to have happened before the end of this RCU Tasks grace period.
950 // current task to a per-CPU list of tasks that rcu_tasks_postscan() must
952 // tasks that have already been removed from the global list of tasks.
998 	 * Idle tasks (or idle injection) within the idle loop are RCU-tasks  in rcu_tasks_is_holdout()
1007 	/* Idle tasks on offline CPUs are RCU-tasks quiescent states. */  in rcu_tasks_is_holdout()
1026 DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
1040 	 * Exiting tasks may escape the tasklist scan. Those are vulnerable  in rcu_tasks_postscan()
1092 /* See if tasks are still holding out, complain if so. */
1112 		pr_err("INFO: rcu_tasks detected stalls on tasks:\n");  in check_holdout_task()
1124 /* Scan the holdout lists for tasks no longer holding out. */
1136 /* Finish off the Tasks-RCU grace period. */
1142 	 * reordering on other CPUs could cause their RCU-tasks read-side  in rcu_tasks_postgp()
1152 	 * In addition, this synchronize_rcu() waits for exiting tasks  in rcu_tasks_postgp()
1155 	 * the final schedule() with TASK_DEAD state to be an RCU TASKS  in rcu_tasks_postgp()
1170 …pr_info("Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tas…  in tasks_rcu_exit_srcu_stall()
1188  * this primitive is intended to determine that all tasks have passed
1201  * synchronize_rcu_tasks - wait until an rcu-tasks grace period has elapsed.
1203  * Control will return to the caller some time after a full rcu-tasks
1205  * executing rcu-tasks read-side critical sections have elapsed.  These
1337 // "Rude" variant of Tasks RCU, inspired by Steve Rostedt's
1353 // Wait for one rude RCU-tasks grace period.
1362 		 "RCU Tasks Rude");
1376  * this primitive is intended to determine that all tasks have passed
1391  * synchronize_rcu_tasks_rude - wait for a rude rcu-tasks grace period
1393  * Control will return to the caller some time after a rude rcu-tasks
1395  * executing rcu-tasks read-side critical sections have elapsed.  These
1453 // Tracing variant of Tasks RCU.  This variant is designed to be used
1472 // flavors, rcu_preempt and rcu_sched.  The fact that RCU Tasks Trace
1482 //	Disables CPU hotplug, adds all currently executing tasks to the
1483 //	holdout list, then checks the state of all tasks that blocked
1484 //	or were preempted within their current RCU Tasks Trace read-side
1489 //	Invokes synchronize_rcu() to wait for late-stage exiting tasks
1499 // The exit_tasks_rcu_finish_trace() synchronizes with exiting tasks.
1529 		 "RCU Tasks Trace");
1559  * Also remove from the per-CPU list of blocked tasks.
1681 		// Check for "running" idle tasks on offline CPUs.  in trc_inspect_reader()
1794 /* Initialize for a new RCU-tasks-trace grace period. */
1833 	// Only after all running tasks have been accounted for is it  in rcu_tasks_trace_pregp_step()
1834 	// safe to take care of the tasks that have blocked within their  in rcu_tasks_trace_pregp_step()
1835 	// current RCU tasks trace read-side critical section.  in rcu_tasks_trace_pregp_step()
1863 	// Wait for late-stage exiting tasks to finish exiting.  in rcu_tasks_trace_postscan()
1868 	// Any tasks that exit after this point will set  in rcu_tasks_trace_postscan()
1872 /* Communicate task state back to the RCU tasks trace stall warning request. */
1891 /* Show the state of a task stalling the current RCU tasks trace GP. */
1899 		pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n");  in show_stalled_task_trace()
1922 /* List stalled IPIs for RCU tasks trace. */
2006  * The callback function will be invoked some time after a trace rcu-tasks
2008  * trace rcu-tasks read-side critical sections have completed. These
2022  * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period
2024  * Control will return to the caller some time after a trace rcu-tasks
2026  * trace rcu-tasks read-side critical sections have elapsed. These read-side
2040 …_held(&rcu_trace_lock_map), "Illegal synchronize_rcu_tasks_trace() in RCU Tasks Trace read-side cr…  in synchronize_rcu_tasks_trace()
2166 	pr_info("Running RCU Tasks wait API self tests\n");  in rcu_tasks_initiate_self_tests()
2173 	pr_info("Running RCU Tasks Rude wait API self tests\n");  in rcu_tasks_initiate_self_tests()
2178 	pr_info("Running RCU Tasks Trace wait API self tests\n");  in rcu_tasks_initiate_self_tests()