1 This is a living document and at times it will be out of date. It is 2 intended to articulate how programming in the Go runtime differs from 3 writing normal Go. It focuses on pervasive concepts rather than 4 details of particular interfaces. 5 6 Scheduler structures 7 ==================== 8 9 The scheduler manages three types of resources that pervade the 10 runtime: Gs, Ms, and Ps. It's important to understand these even if 11 you're not working on the scheduler. 12 13 Gs, Ms, Ps 14 ---------- 15 16 A "G" is simply a goroutine. It's represented by type `g`. When a 17 goroutine exits, its `g` object is returned to a pool of free `g`s and 18 can later be reused for some other goroutine. 19 20 An "M" is an OS thread that can be executing user Go code, runtime 21 code, a system call, or be idle. It's represented by type `m`. There 22 can be any number of Ms at a time since any number of threads may be 23 blocked in system calls. 24 25 Finally, a "P" represents the resources required to execute user Go 26 code, such as scheduler and memory allocator state. It's represented 27 by type `p`. There are exactly `GOMAXPROCS` Ps. A P can be thought of 28 like a CPU in the OS scheduler and the contents of the `p` type like 29 per-CPU state. This is a good place to put state that needs to be 30 sharded for efficiency, but doesn't need to be per-thread or 31 per-goroutine. 32 33 The scheduler's job is to match up a G (the code to execute), an M 34 (where to execute it), and a P (the rights and resources to execute 35 it). When an M stops executing user Go code, for example by entering a 36 system call, it returns its P to the idle P pool. In order to resume 37 executing user Go code, for example on return from a system call, it 38 must acquire a P from the idle pool. 39 40 All `g`, `m`, and `p` objects are heap allocated, but are never freed, 41 so their memory remains type stable. As a result, the runtime can 42 avoid write barriers in the depths of the scheduler. 43 44 User stacks and system stacks 45 ----------------------------- 46 47 Every non-dead G has a *user stack* associated with it, which is what 48 user Go code executes on. User stacks start small (e.g., 2K) and grow 49 or shrink dynamically. 50 51 Every M has a *system stack* associated with it (also known as the M's 52 "g0" stack because it's implemented as a stub G) and, on Unix 53 platforms, a *signal stack* (also known as the M's "gsignal" stack). 54 System and signal stacks cannot grow, but are large enough to execute 55 runtime and cgo code (8K in a pure Go binary; system-allocated in a 56 cgo binary). 57 58 Runtime code often temporarily switches to the system stack using 59 `systemstack`, `mcall`, or `asmcgocall` to perform tasks that must not 60 be preempted, that must not grow the user stack, or that switch user 61 goroutines. Code running on the system stack is implicitly 62 non-preemptible and the garbage collector does not scan system stacks. 63 While running on the system stack, the current user stack is not used 64 for execution. 65 66 `getg()` and `getg().m.curg` 67 ---------------------------- 68 69 To get the current user `g`, use `getg().m.curg`. 70 71 `getg()` alone returns the current `g`, but when executing on the 72 system or signal stacks, this will return the current M's "g0" or 73 "gsignal", respectively. This is usually not what you want. 74 75 To determine if you're running on the user stack or the system stack, 76 use `getg() == getg().m.curg`. 77 78 Error handling and reporting 79 ============================ 80 81 Errors that can reasonably be recovered from in user code should use 82 `panic` like usual. However, there are some situations where `panic` 83 will cause an immediate fatal error, such as when called on the system 84 stack or when called during `mallocgc`. 85 86 Most errors in the runtime are not recoverable. For these, use 87 `throw`, which dumps the traceback and immediately terminates the 88 process. In general, `throw` should be passed a string constant to 89 avoid allocating in perilous situations. By convention, additional 90 details are printed before `throw` using `print` or `println` and the 91 messages are prefixed with "runtime:". 92 93 For runtime error debugging, it's useful to run with 94 `GOTRACEBACK=system` or `GOTRACEBACK=crash`. 95 96 Synchronization 97 =============== 98 99 The runtime has multiple synchronization mechanisms. They differ in 100 semantics and, in particular, in whether they interact with the 101 goroutine scheduler or the OS scheduler. 102 103 The simplest is `mutex`, which is manipulated using `lock` and 104 `unlock`. This should be used to protect shared structures for short 105 periods. Blocking on a `mutex` directly blocks the M, without 106 interacting with the Go scheduler. This means it is safe to use from 107 the lowest levels of the runtime, but also prevents any associated G 108 and P from being rescheduled. `rwmutex` is similar. 109 110 For one-shot notifications, use `note`, which provides `notesleep` and 111 `notewakeup`. Unlike traditional UNIX `sleep`/`wakeup`, `note`s are 112 race-free, so `notesleep` returns immediately if the `notewakeup` has 113 already happened. A `note` can be reset after use with `noteclear`, 114 which must not race with a sleep or wakeup. Like `mutex`, blocking on 115 a `note` blocks the M. However, there are different ways to sleep on a 116 `note`:`notesleep` also prevents rescheduling of any associated G and 117 P, while `notetsleepg` acts like a blocking system call that allows 118 the P to be reused to run another G. This is still less efficient than 119 blocking the G directly since it consumes an M. 120 121 To interact directly with the goroutine scheduler, use `gopark` and 122 `goready`. `gopark` parks the current goroutineputting it in the 123 "waiting" state and removing it from the scheduler's run queueand 124 schedules another goroutine on the current M/P. `goready` puts a 125 parked goroutine back in the "runnable" state and adds it to the run 126 queue. 127 128 In summary, 129 130 <table> 131 <tr><th></th><th colspan="3">Blocks</th></tr> 132 <tr><th>Interface</th><th>G</th><th>M</th><th>P</th></tr> 133 <tr><td>(rw)mutex</td><td>Y</td><td>Y</td><td>Y</td></tr> 134 <tr><td>note</td><td>Y</td><td>Y</td><td>Y/N</td></tr> 135 <tr><td>park</td><td>Y</td><td>N</td><td>N</td></tr> 136 </table> 137 138 Unmanaged memory 139 ================ 140 141 In general, the runtime tries to use regular heap allocation. However, 142 in some cases the runtime must allocate objects outside of the garbage 143 collected heap, in *unmanaged memory*. This is necessary if the 144 objects are part of the memory manager itself or if they must be 145 allocated in situations where the caller may not have a P. 146 147 There are three mechanisms for allocating unmanaged memory: 148 149 * sysAlloc obtains memory directly from the OS. This comes in whole 150 multiples of the system page size, but it can be freed with sysFree. 151 152 * persistentalloc combines multiple smaller allocations into a single 153 sysAlloc to avoid fragmentation. However, there is no way to free 154 persistentalloced objects (hence the name). 155 156 * fixalloc is a SLAB-style allocator that allocates objects of a fixed 157 size. fixalloced objects can be freed, but this memory can only be 158 reused by the same fixalloc pool, so it can only be reused for 159 objects of the same type. 160 161 In general, types that are allocated using any of these should be 162 marked `//go:notinheap` (see below). 163 164 Objects that are allocated in unmanaged memory **must not** contain 165 heap pointers unless the following rules are also obeyed: 166 167 1. Any pointers from unmanaged memory to the heap must be added as 168 explicit garbage collection roots in `runtime.markroot`. 169 170 2. If the memory is reused, the heap pointers must be zero-initialized 171 before they become visible as GC roots. Otherwise, the GC may 172 observe stale heap pointers. See "Zero-initialization versus 173 zeroing". 174 175 Zero-initialization versus zeroing 176 ================================== 177 178 There are two types of zeroing in the runtime, depending on whether 179 the memory is already initialized to a type-safe state. 180 181 If memory is not in a type-safe state, meaning it potentially contains 182 "garbage" because it was just allocated and it is being initialized 183 for first use, then it must be *zero-initialized* using 184 `memclrNoHeapPointers` or non-pointer writes. This does not perform 185 write barriers. 186 187 If memory is already in a type-safe state and is simply being set to 188 the zero value, this must be done using regular writes, `typedmemclr`, 189 or `memclrHasPointers`. This performs write barriers. 190 191 Runtime-only compiler directives 192 ================================ 193 194 In addition to the "//go:" directives documented in "go doc compile", 195 the compiler supports additional directives only in the runtime. 196 197 go:systemstack 198 -------------- 199 200 `go:systemstack` indicates that a function must run on the system 201 stack. This is checked dynamically by a special function prologue. 202 203 go:nowritebarrier 204 ----------------- 205 206 `go:nowritebarrier` directs the compiler to emit an error if the 207 following function contains any write barriers. (It *does not* 208 suppress the generation of write barriers; it is simply an assertion.) 209 210 Usually you want `go:nowritebarrierrec`. `go:nowritebarrier` is 211 primarily useful in situations where it's "nice" not to have write 212 barriers, but not required for correctness. 213 214 go:nowritebarrierrec and go:yeswritebarrierrec 215 ---------------------------------------------- 216 217 `go:nowritebarrierrec` directs the compiler to emit an error if the 218 following function or any function it calls recursively, up to a 219 `go:yeswritebarrierrec`, contains a write barrier. 220 221 Logically, the compiler floods the call graph starting from each 222 `go:nowritebarrierrec` function and produces an error if it encounters 223 a function containing a write barrier. This flood stops at 224 `go:yeswritebarrierrec` functions. 225 226 `go:nowritebarrierrec` is used in the implementation of the write 227 barrier to prevent infinite loops. 228 229 Both directives are used in the scheduler. The write barrier requires 230 an active P (`getg().m.p != nil`) and scheduler code often runs 231 without an active P. In this case, `go:nowritebarrierrec` is used on 232 functions that release the P or may run without a P and 233 `go:yeswritebarrierrec` is used when code re-acquires an active P. 234 Since these are function-level annotations, code that releases or 235 acquires a P may need to be split across two functions. 236 237 go:notinheap 238 ------------ 239 240 `go:notinheap` applies to type declarations. It indicates that a type 241 must never be allocated from the GC'd heap. Specifically, pointers to 242 this type must always fail the `runtime.inheap` check. The type may be 243 used for global variables, for stack variables, or for objects in 244 unmanaged memory (e.g., allocated with `sysAlloc`, `persistentalloc`, 245 `fixalloc`, or from a manually-managed span). Specifically: 246 247 1. `new(T)`, `make([]T)`, `append([]T, ...)` and implicit heap 248 allocation of T are disallowed. (Though implicit allocations are 249 disallowed in the runtime anyway.) 250 251 2. A pointer to a regular type (other than `unsafe.Pointer`) cannot be 252 converted to a pointer to a `go:notinheap` type, even if they have 253 the same underlying type. 254 255 3. Any type that contains a `go:notinheap` type is itself 256 `go:notinheap`. Structs and arrays are `go:notinheap` if their 257 elements are. Maps and channels of `go:notinheap` types are 258 disallowed. To keep things explicit, any type declaration where the 259 type is implicitly `go:notinheap` must be explicitly marked 260 `go:notinheap` as well. 261 262 4. Write barriers on pointers to `go:notinheap` types can be omitted. 263 264 The last point is the real benefit of `go:notinheap`. The runtime uses 265 it for low-level internal structures to avoid memory barriers in the 266 scheduler and the memory allocator where they are illegal or simply 267 inefficient. This mechanism is reasonably safe and does not compromise 268 the readability of the runtime. 269
