blob: 37eca00796eeebf805afbbf1504753c1a0febec0 [file] [log] [blame]
Tejun Heo70c83e12005-09-11 09:37:19 +09001
2SCSI EH
3======================================
4
5 This document describes SCSI midlayer error handling infrastructure.
6Please refer to Documentation/scsi/scsi_mid_low_api.txt for more
7information regarding SCSI midlayer.
8
9TABLE OF CONTENTS
10
11[1] How SCSI commands travel through the midlayer and to EH
12 [1-1] struct scsi_cmnd
13 [1-2] How do scmd's get completed?
14 [1-2-1] Completing a scmd w/ scsi_done
15 [1-2-2] Completing a scmd w/ timeout
16 [1-3] How EH takes over
17[2] How SCSI EH works
18 [2-1] EH through fine-grained callbacks
19 [2-1-1] Overview
20 [2-1-2] Flow of scmds through EH
21 [2-1-3] Flow of control
Christoph Hellwig9227c332006-04-01 19:21:04 +020022 [2-2] EH through transportt->eh_strategy_handler()
23 [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
24 [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +090025 [2-2-3] Things to consider
26
27
28[1] How SCSI commands travel through the midlayer and to EH
29
30[1-1] struct scsi_cmnd
31
32 Each SCSI command is represented with struct scsi_cmnd (== scmd). A
33scmd has two list_head's to link itself into lists. The two are
34scmd->list and scmd->eh_entry. The former is used for free list or
35per-device allocated scmd list and not of much interest to this EH
36discussion. The latter is used for completion and EH lists and unless
37otherwise stated scmds are always linked using scmd->eh_entry in this
38discussion.
39
40
41[1-2] How do scmd's get completed?
42
43 Once LLDD gets hold of a scmd, either the LLDD will complete the
44command by calling scsi_done callback passed from midlayer when
Hannes Reinecke6ad55502013-11-11 13:44:57 +010045invoking hostt->queuecommand() or the block layer will time it out.
Tejun Heo70c83e12005-09-11 09:37:19 +090046
47
48[1-2-1] Completing a scmd w/ scsi_done
49
50 For all non-EH commands, scsi_done() is the completion callback. It
Hannes Reinecke6ad55502013-11-11 13:44:57 +010051just calls blk_complete_request() to delete the block layer timer and
52raise SCSI_SOFTIRQ
Tejun Heo70c83e12005-09-11 09:37:19 +090053
54 SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to
55determine what to do with the command. scsi_decide_disposition()
56looks at the scmd->result value and sense data to determine what to do
57with the command.
58
59 - SUCCESS
60 scsi_finish_command() is invoked for the command. The
Hannes Reinecke6ad55502013-11-11 13:44:57 +010061 function does some maintenance chores and then calls
62 scsi_io_completion() to finish the I/O.
63 scsi_io_completion() then notifies the block layer on
64 the completed request by calling blk_end_request and
65 friends or figures out what to do with the remainder
66 of the data in case of an error.
Tejun Heo70c83e12005-09-11 09:37:19 +090067
68 - NEEDS_RETRY
69 - ADD_TO_MLQUEUE
70 scmd is requeued to blk queue.
71
72 - otherwise
73 scsi_eh_scmd_add(scmd, 0) is invoked for the command. See
Matt LaPlante5d3f0832006-11-30 05:21:10 +010074 [1-3] for details of this function.
Tejun Heo70c83e12005-09-11 09:37:19 +090075
76
77[1-2-2] Completing a scmd w/ timeout
78
79 The timeout handler is scsi_times_out(). When a timeout occurs, this
80function
81
Stefan Richter8c0ae6562005-11-05 01:35:05 +010082 1. invokes optional hostt->eh_timed_out() callback. Return value can
Tejun Heo70c83e12005-09-11 09:37:19 +090083 be one of
84
Hannes Reinecke6ad55502013-11-11 13:44:57 +010085 - BLK_EH_HANDLED
86 This indicates that eh_timed_out() dealt with the timeout.
87 The command is passed back to the block layer and completed
88 via __blk_complete_requests().
Tejun Heo70c83e12005-09-11 09:37:19 +090089
Hannes Reinecke6ad55502013-11-11 13:44:57 +010090 *NOTE* After returning BLK_EH_HANDLED the SCSI layer is
91 assumed to be finished with the command, and no other
92 functions from the SCSI layer will be called. So this
93 should typically only be returned if the eh_timed_out()
94 handler raced with normal completion.
95
96 - BLK_EH_RESET_TIMER
Tejun Heo70c83e12005-09-11 09:37:19 +090097 This indicates that more time is required to finish the
98 command. Timer is restarted. This action is counted as a
99 retry and only allowed scmd->allowed + 1(!) times. Once the
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100100 limit is reached, action for BLK_EH_NOT_HANDLED is taken instead.
Tejun Heo70c83e12005-09-11 09:37:19 +0900101
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100102 - BLK_EH_NOT_HANDLED
103 eh_timed_out() callback did not handle the command.
Tejun Heo70c83e12005-09-11 09:37:19 +0900104 Step #2 is taken.
105
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100106 2. If the host supports asynchronous completion (as indicated by the
107 no_async_abort setting in the host template) scsi_abort_command()
108 is invoked to schedule an asynchrous abort. If that fails
109 Step #3 is taken.
110
Tejun Heo70c83e12005-09-11 09:37:19 +0900111 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the
112 command. See [1-3] for more information.
113
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100114[1-3] Asynchronous command aborts
Tejun Heo70c83e12005-09-11 09:37:19 +0900115
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100116 After a timeout occurs a command abort is scheduled from
117 scsi_abort_command(). If the abort is successful the command
118 will either be retried (if the number of retries is not exhausted)
119 or terminated with DID_TIME_OUT.
120 Otherwise scsi_eh_scmd_add() is invoked for the command.
121 See [1-4] for more information.
122
123[1-4] How EH takes over
Tejun Heo70c83e12005-09-11 09:37:19 +0900124
125 scmds enter EH via scsi_eh_scmd_add(), which does the following.
126
127 1. Turns on scmd->eh_eflags as requested. It's 0 for error
128 completions and SCSI_EH_CANCEL_CMD for timeouts.
129
130 2. Links scmd->eh_entry to shost->eh_cmd_q
131
132 3. Sets SHOST_RECOVERY bit in shost->shost_state
133
134 4. Increments shost->host_failed
135
136 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
137
138 As can be seen above, once any scmd is added to shost->eh_cmd_q,
139SHOST_RECOVERY shost_state bit is turned on. This prevents any new
140scmd to be issued from blk queue to the host; eventually, all scmds on
141the host either complete normally, fail and get added to eh_cmd_q, or
142time out and get added to shost->eh_cmd_q.
143
144 If all scmds either complete or fail, the number of in-flight scmds
145becomes equal to the number of failed scmds - i.e. shost->host_busy ==
146shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
147SCSI EH thread can expect that all in-flight commands have failed and
148are linked on shost->eh_cmd_q.
149
150 Note that this does not mean lower layers are quiescent. If a LLDD
151completed a scmd with error status, the LLDD and lower layers are
152assumed to forget about the scmd at that point. However, if a scmd
Stefan Richter8c0ae6562005-11-05 01:35:05 +0100153has timed out, unless hostt->eh_timed_out() made lower layers forget
Tejun Heo70c83e12005-09-11 09:37:19 +0900154about the scmd, which currently no LLDD does, the command is still
155active as long as lower layers are concerned and completion could
156occur at any time. Of course, all such completions are ignored as the
157timer has already expired.
158
159 We'll talk about how SCSI EH takes actions to abort - make LLDD
160forget about - timed out scmds later.
161
162
163[2] How SCSI EH works
164
165 LLDD's can implement SCSI EH actions in one of the following two
166ways.
167
168 - Fine-grained EH callbacks
169 LLDD can implement fine-grained EH callbacks and let SCSI
170 midlayer drive error handling and call appropriate callbacks.
Matt LaPlantefff92892006-10-03 22:47:42 +0200171 This will be discussed further in [2-1].
Tejun Heo70c83e12005-09-11 09:37:19 +0900172
173 - eh_strategy_handler() callback
174 This is one big callback which should perform whole error
Finn Thain542cb452014-10-03 11:42:17 +1000175 handling. As such, it should do all chores the SCSI midlayer
Tejun Heo70c83e12005-09-11 09:37:19 +0900176 performs during recovery. This will be discussed in [2-2].
177
178 Once recovery is complete, SCSI EH resumes normal operation by
179calling scsi_restart_operations(), which
180
181 1. Checks if door locking is needed and locks door.
182
183 2. Clears SHOST_RECOVERY shost_state bit
184
185 3. Wakes up waiters on shost->host_wait. This occurs if someone
186 calls scsi_block_when_processing_errors() on the host.
187 (*QUESTION* why is it needed? All operations will be blocked
188 anyway after it reaches blk queue.)
189
190 4. Kicks queues in all devices on the host in the asses
191
192
193[2-1] EH through fine-grained callbacks
194
195[2-1-1] Overview
196
197 If eh_strategy_handler() is not present, SCSI midlayer takes charge
198of driving error handling. EH's goals are two - make LLDD, host and
199device forget about timed out scmds and make them ready for new
200commands. A scmd is said to be recovered if the scmd is forgotten by
201lower layers and lower layers are ready to process or fail the scmd
202again.
203
204 To achieve these goals, EH performs recovery actions with increasing
Matt LaPlante2fe0ae72006-10-03 22:50:39 +0200205severity. Some actions are performed by issuing SCSI commands and
Tejun Heo70c83e12005-09-11 09:37:19 +0900206others are performed by invoking one of the following fine-grained
207hostt EH callbacks. Callbacks may be omitted and omitted ones are
208considered to fail always.
209
210int (* eh_abort_handler)(struct scsi_cmnd *);
211int (* eh_device_reset_handler)(struct scsi_cmnd *);
212int (* eh_bus_reset_handler)(struct scsi_cmnd *);
213int (* eh_host_reset_handler)(struct scsi_cmnd *);
214
215 Higher-severity actions are taken only when lower-severity actions
216cannot recover some of failed scmds. Also, note that failure of the
217highest-severity action means EH failure and results in offlining of
218all unrecovered devices.
219
220 During recovery, the following rules are followed
221
222 - Recovery actions are performed on failed scmds on the to do list,
223 eh_work_q. If a recovery action succeeds for a scmd, recovered
224 scmds are removed from eh_work_q.
225
226 Note that single recovery action on a scmd can recover multiple
227 scmds. e.g. resetting a device recovers all failed scmds on the
228 device.
229
230 - Higher severity actions are taken iff eh_work_q is not empty after
231 lower severity actions are complete.
232
233 - EH reuses failed scmds to issue commands for recovery. For
234 timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
235 before reusing it for EH commands.
236
237 When a scmd is recovered, the scmd is moved from eh_work_q to EH
238local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
239recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
240either retry or error-finish (notify upper layer of failure) recovered
241scmds.
242
243 scmds are retried iff its sdev is still online (not offlined during
244EH), REQ_FAILFAST is not set and ++scmd->retries is less than
245scmd->allowed.
246
247
248[2-1-2] Flow of scmds through EH
249
250 1. Error completion / time out
251 ACTION: scsi_eh_scmd_add() is invoked for scmd
252 - set scmd->eh_eflags
253 - add scmd to shost->eh_cmd_q
254 - set SHOST_RECOVERY
255 - shost->host_failed++
256 LOCKING: shost->host_lock
257
258 2. EH starts
259 ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
260 is cleared.
261 LOCKING: shost->host_lock (not strictly necessary, just for
262 consistency)
263
264 3. scmd recovered
265 ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
Tejun Heo70c83e12005-09-11 09:37:19 +0900266 - clear scmd->eh_eflags
267 - scsi_setup_cmd_retry()
268 - move from local eh_work_q to local eh_done_q
269 LOCKING: none
Wei Fang72d8c362016-06-07 14:53:56 +0800270 CONCURRENCY: at most one thread per separate eh_work_q to
271 keep queue manipulation lockless
Tejun Heo70c83e12005-09-11 09:37:19 +0900272
273 4. EH completes
274 ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
Wei Fang72d8c362016-06-07 14:53:56 +0800275 layer of failure. May be called concurrently but must have
276 a no more than one thread per separate eh_work_q to
277 manipulate the queue locklessly
Tejun Heo70c83e12005-09-11 09:37:19 +0900278 - scmd is removed from eh_done_q and scmd->eh_entry is cleared
279 - if retry is necessary, scmd is requeued using
280 scsi_queue_insert()
281 - otherwise, scsi_finish_command() is invoked for scmd
Wei Fang72d8c362016-06-07 14:53:56 +0800282 - zero shost->host_failed
Tejun Heo70c83e12005-09-11 09:37:19 +0900283 LOCKING: queue or finish function performs appropriate locking
284
285
286[2-1-3] Flow of control
287
288 EH through fine-grained callbacks start from scsi_unjam_host().
289
290<<scsi_unjam_host>>
291
292 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
293 eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
294 cleared by this action.
295
296 2. Invoke scsi_eh_get_sense.
297
298 <<scsi_eh_get_sense>>
299
300 This action is taken for each error-completed
301 (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most
302 SCSI transports/LLDDs automatically acquire sense data on
303 command failures (autosense). Autosense is recommended for
304 performance reasons and as sense information could get out of
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300305 sync between occurrence of CHECK CONDITION and this action.
Tejun Heo70c83e12005-09-11 09:37:19 +0900306
307 Note that if autosense is not supported, scmd->sense_buffer
308 contains invalid sense data when error-completing the scmd
309 with scsi_done(). scsi_decide_disposition() always returns
310 FAILED in such cases thus invoking SCSI EH. When the scmd
311 reaches here, sense data is acquired and
312 scsi_decide_disposition() is called again.
313
314 1. Invoke scsi_request_sense() which issues REQUEST_SENSE
315 command. If fails, no action. Note that taking no action
316 causes higher-severity recovery to be taken for the scmd.
317
318 2. Invoke scsi_decide_disposition() on the scmd
319
320 - SUCCESS
321 scmd->retries is set to scmd->allowed preventing
322 scsi_eh_flush_done_q() from retrying the scmd and
323 scsi_eh_finish_cmd() is invoked.
324
325 - NEEDS_RETRY
326 scsi_eh_finish_cmd() invoked
327
328 - otherwise
329 No action.
330
331 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
332
333 <<scsi_eh_abort_cmds>>
334
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100335 This action is taken for each timed out command when
336 no_async_abort is enabled in the host template.
Tejun Heo70c83e12005-09-11 09:37:19 +0900337 hostt->eh_abort_handler() is invoked for each scmd. The
338 handler returns SUCCESS if it has succeeded to make LLDD and
339 all related hardware forget about the scmd.
340
341 If a timedout scmd is successfully aborted and the sdev is
342 either offline or ready, scsi_eh_finish_cmd() is invoked for
343 the scmd. Otherwise, the scmd is left in eh_work_q for
344 higher-severity actions.
345
346 Note that both offline and ready status mean that the sdev is
347 ready to process new scmds, where processing also implies
348 immediate failing; thus, if a sdev is in one of the two
349 states, no further recovery action is needed.
350
351 Device readiness is tested using scsi_eh_tur() which issues
352 TEST_UNIT_READY command. Note that the scmd must have been
353 aborted successfully before reusing it for TEST_UNIT_READY.
354
355 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
356
357 <<scsi_eh_ready_devs>>
358
359 This function takes four increasingly more severe measures to
360 make failed sdevs ready for new commands.
361
362 1. Invoke scsi_eh_stu()
363
364 <<scsi_eh_stu>>
365
366 For each sdev which has failed scmds with valid sense data
367 of which scsi_check_sense()'s verdict is FAILED,
368 START_STOP_UNIT command is issued w/ start=1. Note that
369 as we explicitly choose error-completed scmds, it is known
370 that lower layers have forgotten about the scmd and we can
371 reuse it for STU.
372
373 If STU succeeds and the sdev is either offline or ready,
374 all failed scmds on the sdev are EH-finished with
375 scsi_eh_finish_cmd().
376
377 *NOTE* If hostt->eh_abort_handler() isn't implemented or
378 failed, we may still have timed out scmds at this point
379 and STU doesn't make lower layers forget about those
380 scmds. Yet, this function EH-finish all scmds on the sdev
381 if STU succeeds leaving lower layers in an inconsistent
382 state. It seems that STU action should be taken only when
383 a sdev has no timed out scmd.
384
385 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
386
387 <<scsi_eh_bus_device_reset>>
388
389 This action is very similar to scsi_eh_stu() except that,
390 instead of issuing STU, hostt->eh_device_reset_handler()
391 is used. Also, as we're not issuing SCSI commands and
392 resetting clears all scmds on the sdev, there is no need
393 to choose error-completed scmds.
394
395 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
396
397 <<scsi_eh_bus_reset>>
398
399 hostt->eh_bus_reset_handler() is invoked for each channel
400 with failed scmds. If bus reset succeeds, all failed
401 scmds on all ready or offline sdevs on the channel are
402 EH-finished.
403
404 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
405
406 <<scsi_eh_host_reset>>
407
408 This is the last resort. hostt->eh_host_reset_handler()
409 is invoked. If host reset succeeds, all failed scmds on
410 all ready or offline sdevs on the host are EH-finished.
411
412 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
413
414 <<scsi_eh_offline_sdevs>>
415
416 Take all sdevs which still have unrecovered scmds offline
417 and EH-finish the scmds.
418
419 5. Invoke scsi_eh_flush_done_q().
420
421 <<scsi_eh_flush_done_q>>
422
423 At this point all scmds are recovered (or given up) and
424 put on eh_done_q by scsi_eh_finish_cmd(). This function
425 flushes eh_done_q by either retrying or notifying upper
426 layer of failure of the scmds.
427
428
Christoph Hellwig9227c332006-04-01 19:21:04 +0200429[2-2] EH through transportt->eh_strategy_handler()
Tejun Heo70c83e12005-09-11 09:37:19 +0900430
Christoph Hellwig9227c332006-04-01 19:21:04 +0200431 transportt->eh_strategy_handler() is invoked in the place of
Tejun Heo70c83e12005-09-11 09:37:19 +0900432scsi_unjam_host() and it is responsible for whole recovery process.
433On completion, the handler should have made lower layers forget about
434all failed scmds and either ready for new commands or offline. Also,
Finn Thain542cb452014-10-03 11:42:17 +1000435it should perform SCSI EH maintenance chores to maintain integrity of
Tejun Heo70c83e12005-09-11 09:37:19 +0900436SCSI midlayer. IOW, of the steps described in [2-1-2], all steps
437except for #1 must be implemented by eh_strategy_handler().
438
439
Christoph Hellwig9227c332006-04-01 19:21:04 +0200440[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +0900441
442 The following conditions are true on entry to the handler.
443
444 - Each failed scmd's eh_flags field is set appropriately.
445
446 - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
447
448 - SHOST_RECOVERY is set.
449
450 - shost->host_failed == shost->host_busy
451
452
Christoph Hellwig9227c332006-04-01 19:21:04 +0200453[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +0900454
455 The following conditions must be true on exit from the handler.
456
457 - shost->host_failed is zero.
458
459 - Each scmd's eh_eflags field is cleared.
460
461 - Each scmd is in such a state that scsi_setup_cmd_retry() on the
462 scmd doesn't make any difference.
463
464 - shost->eh_cmd_q is cleared.
465
466 - Each scmd->eh_entry is cleared.
467
468 - Either scsi_queue_insert() or scsi_finish_command() is called on
469 each scmd. Note that the handler is free to use scmd->retries and
470 ->allowed to limit the number of retries.
471
472
473[2-2-3] Things to consider
474
475 - Know that timed out scmds are still active on lower layers. Make
476 lower layers forget about them before doing anything else with
477 those scmds.
478
479 - For consistency, when accessing/modifying shost data structure,
480 grab shost->host_lock.
481
482 - On completion, each failed sdev must have forgotten about all
483 active scmds.
484
485 - On completion, each failed sdev must be ready for new commands or
486 offline.
487
488
489--
490Tejun Heo
491htejun@gmail.com
49211th September 2005