blob: 8638f61c8c9db70d767302497d417be6c9a0a030 [file] [log] [blame]
Tejun Heo70c83e12005-09-11 09:37:19 +09001
2SCSI EH
3======================================
4
5 This document describes SCSI midlayer error handling infrastructure.
6Please refer to Documentation/scsi/scsi_mid_low_api.txt for more
7information regarding SCSI midlayer.
8
9TABLE OF CONTENTS
10
11[1] How SCSI commands travel through the midlayer and to EH
12 [1-1] struct scsi_cmnd
13 [1-2] How do scmd's get completed?
14 [1-2-1] Completing a scmd w/ scsi_done
15 [1-2-2] Completing a scmd w/ timeout
16 [1-3] How EH takes over
17[2] How SCSI EH works
18 [2-1] EH through fine-grained callbacks
19 [2-1-1] Overview
20 [2-1-2] Flow of scmds through EH
21 [2-1-3] Flow of control
Christoph Hellwig9227c332006-04-01 19:21:04 +020022 [2-2] EH through transportt->eh_strategy_handler()
23 [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
24 [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +090025 [2-2-3] Things to consider
26
27
28[1] How SCSI commands travel through the midlayer and to EH
29
30[1-1] struct scsi_cmnd
31
32 Each SCSI command is represented with struct scsi_cmnd (== scmd). A
33scmd has two list_head's to link itself into lists. The two are
34scmd->list and scmd->eh_entry. The former is used for free list or
35per-device allocated scmd list and not of much interest to this EH
36discussion. The latter is used for completion and EH lists and unless
37otherwise stated scmds are always linked using scmd->eh_entry in this
38discussion.
39
40
41[1-2] How do scmd's get completed?
42
43 Once LLDD gets hold of a scmd, either the LLDD will complete the
44command by calling scsi_done callback passed from midlayer when
Hannes Reinecke6ad55502013-11-11 13:44:57 +010045invoking hostt->queuecommand() or the block layer will time it out.
Tejun Heo70c83e12005-09-11 09:37:19 +090046
47
48[1-2-1] Completing a scmd w/ scsi_done
49
50 For all non-EH commands, scsi_done() is the completion callback. It
Hannes Reinecke6ad55502013-11-11 13:44:57 +010051just calls blk_complete_request() to delete the block layer timer and
52raise SCSI_SOFTIRQ
Tejun Heo70c83e12005-09-11 09:37:19 +090053
54 SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to
55determine what to do with the command. scsi_decide_disposition()
56looks at the scmd->result value and sense data to determine what to do
57with the command.
58
59 - SUCCESS
60 scsi_finish_command() is invoked for the command. The
Hannes Reinecke6ad55502013-11-11 13:44:57 +010061 function does some maintenance chores and then calls
62 scsi_io_completion() to finish the I/O.
63 scsi_io_completion() then notifies the block layer on
64 the completed request by calling blk_end_request and
65 friends or figures out what to do with the remainder
66 of the data in case of an error.
Tejun Heo70c83e12005-09-11 09:37:19 +090067
68 - NEEDS_RETRY
69 - ADD_TO_MLQUEUE
70 scmd is requeued to blk queue.
71
72 - otherwise
73 scsi_eh_scmd_add(scmd, 0) is invoked for the command. See
Matt LaPlante5d3f0832006-11-30 05:21:10 +010074 [1-3] for details of this function.
Tejun Heo70c83e12005-09-11 09:37:19 +090075
76
77[1-2-2] Completing a scmd w/ timeout
78
79 The timeout handler is scsi_times_out(). When a timeout occurs, this
80function
81
Stefan Richter8c0ae6562005-11-05 01:35:05 +010082 1. invokes optional hostt->eh_timed_out() callback. Return value can
Tejun Heo70c83e12005-09-11 09:37:19 +090083 be one of
84
Hannes Reinecke6ad55502013-11-11 13:44:57 +010085 - BLK_EH_HANDLED
86 This indicates that eh_timed_out() dealt with the timeout.
87 The command is passed back to the block layer and completed
88 via __blk_complete_requests().
Tejun Heo70c83e12005-09-11 09:37:19 +090089
Hannes Reinecke6ad55502013-11-11 13:44:57 +010090 *NOTE* After returning BLK_EH_HANDLED the SCSI layer is
91 assumed to be finished with the command, and no other
92 functions from the SCSI layer will be called. So this
93 should typically only be returned if the eh_timed_out()
94 handler raced with normal completion.
95
96 - BLK_EH_RESET_TIMER
Tejun Heo70c83e12005-09-11 09:37:19 +090097 This indicates that more time is required to finish the
98 command. Timer is restarted. This action is counted as a
99 retry and only allowed scmd->allowed + 1(!) times. Once the
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100100 limit is reached, action for BLK_EH_NOT_HANDLED is taken instead.
Tejun Heo70c83e12005-09-11 09:37:19 +0900101
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100102 - BLK_EH_NOT_HANDLED
103 eh_timed_out() callback did not handle the command.
Tejun Heo70c83e12005-09-11 09:37:19 +0900104 Step #2 is taken.
105
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100106 2. If the host supports asynchronous completion (as indicated by the
107 no_async_abort setting in the host template) scsi_abort_command()
108 is invoked to schedule an asynchrous abort. If that fails
109 Step #3 is taken.
110
Tejun Heo70c83e12005-09-11 09:37:19 +0900111 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the
112 command. See [1-3] for more information.
113
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100114[1-3] Asynchronous command aborts
Tejun Heo70c83e12005-09-11 09:37:19 +0900115
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100116 After a timeout occurs a command abort is scheduled from
117 scsi_abort_command(). If the abort is successful the command
118 will either be retried (if the number of retries is not exhausted)
119 or terminated with DID_TIME_OUT.
120 Otherwise scsi_eh_scmd_add() is invoked for the command.
121 See [1-4] for more information.
122
123[1-4] How EH takes over
Tejun Heo70c83e12005-09-11 09:37:19 +0900124
125 scmds enter EH via scsi_eh_scmd_add(), which does the following.
126
127 1. Turns on scmd->eh_eflags as requested. It's 0 for error
128 completions and SCSI_EH_CANCEL_CMD for timeouts.
129
130 2. Links scmd->eh_entry to shost->eh_cmd_q
131
132 3. Sets SHOST_RECOVERY bit in shost->shost_state
133
134 4. Increments shost->host_failed
135
136 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
137
138 As can be seen above, once any scmd is added to shost->eh_cmd_q,
139SHOST_RECOVERY shost_state bit is turned on. This prevents any new
140scmd to be issued from blk queue to the host; eventually, all scmds on
141the host either complete normally, fail and get added to eh_cmd_q, or
142time out and get added to shost->eh_cmd_q.
143
144 If all scmds either complete or fail, the number of in-flight scmds
145becomes equal to the number of failed scmds - i.e. shost->host_busy ==
146shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
147SCSI EH thread can expect that all in-flight commands have failed and
148are linked on shost->eh_cmd_q.
149
150 Note that this does not mean lower layers are quiescent. If a LLDD
151completed a scmd with error status, the LLDD and lower layers are
152assumed to forget about the scmd at that point. However, if a scmd
Stefan Richter8c0ae6562005-11-05 01:35:05 +0100153has timed out, unless hostt->eh_timed_out() made lower layers forget
Tejun Heo70c83e12005-09-11 09:37:19 +0900154about the scmd, which currently no LLDD does, the command is still
155active as long as lower layers are concerned and completion could
156occur at any time. Of course, all such completions are ignored as the
157timer has already expired.
158
159 We'll talk about how SCSI EH takes actions to abort - make LLDD
160forget about - timed out scmds later.
161
162
163[2] How SCSI EH works
164
165 LLDD's can implement SCSI EH actions in one of the following two
166ways.
167
168 - Fine-grained EH callbacks
169 LLDD can implement fine-grained EH callbacks and let SCSI
170 midlayer drive error handling and call appropriate callbacks.
Matt LaPlantefff92892006-10-03 22:47:42 +0200171 This will be discussed further in [2-1].
Tejun Heo70c83e12005-09-11 09:37:19 +0900172
173 - eh_strategy_handler() callback
174 This is one big callback which should perform whole error
Finn Thain542cb452014-10-03 11:42:17 +1000175 handling. As such, it should do all chores the SCSI midlayer
Tejun Heo70c83e12005-09-11 09:37:19 +0900176 performs during recovery. This will be discussed in [2-2].
177
178 Once recovery is complete, SCSI EH resumes normal operation by
179calling scsi_restart_operations(), which
180
181 1. Checks if door locking is needed and locks door.
182
183 2. Clears SHOST_RECOVERY shost_state bit
184
185 3. Wakes up waiters on shost->host_wait. This occurs if someone
186 calls scsi_block_when_processing_errors() on the host.
187 (*QUESTION* why is it needed? All operations will be blocked
188 anyway after it reaches blk queue.)
189
190 4. Kicks queues in all devices on the host in the asses
191
192
193[2-1] EH through fine-grained callbacks
194
195[2-1-1] Overview
196
197 If eh_strategy_handler() is not present, SCSI midlayer takes charge
198of driving error handling. EH's goals are two - make LLDD, host and
199device forget about timed out scmds and make them ready for new
200commands. A scmd is said to be recovered if the scmd is forgotten by
201lower layers and lower layers are ready to process or fail the scmd
202again.
203
204 To achieve these goals, EH performs recovery actions with increasing
Matt LaPlante2fe0ae72006-10-03 22:50:39 +0200205severity. Some actions are performed by issuing SCSI commands and
Tejun Heo70c83e12005-09-11 09:37:19 +0900206others are performed by invoking one of the following fine-grained
207hostt EH callbacks. Callbacks may be omitted and omitted ones are
208considered to fail always.
209
210int (* eh_abort_handler)(struct scsi_cmnd *);
211int (* eh_device_reset_handler)(struct scsi_cmnd *);
212int (* eh_bus_reset_handler)(struct scsi_cmnd *);
213int (* eh_host_reset_handler)(struct scsi_cmnd *);
214
215 Higher-severity actions are taken only when lower-severity actions
216cannot recover some of failed scmds. Also, note that failure of the
217highest-severity action means EH failure and results in offlining of
218all unrecovered devices.
219
220 During recovery, the following rules are followed
221
222 - Recovery actions are performed on failed scmds on the to do list,
223 eh_work_q. If a recovery action succeeds for a scmd, recovered
224 scmds are removed from eh_work_q.
225
226 Note that single recovery action on a scmd can recover multiple
227 scmds. e.g. resetting a device recovers all failed scmds on the
228 device.
229
230 - Higher severity actions are taken iff eh_work_q is not empty after
231 lower severity actions are complete.
232
233 - EH reuses failed scmds to issue commands for recovery. For
234 timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
235 before reusing it for EH commands.
236
237 When a scmd is recovered, the scmd is moved from eh_work_q to EH
238local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
239recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
240either retry or error-finish (notify upper layer of failure) recovered
241scmds.
242
243 scmds are retried iff its sdev is still online (not offlined during
244EH), REQ_FAILFAST is not set and ++scmd->retries is less than
245scmd->allowed.
246
247
248[2-1-2] Flow of scmds through EH
249
250 1. Error completion / time out
251 ACTION: scsi_eh_scmd_add() is invoked for scmd
252 - set scmd->eh_eflags
253 - add scmd to shost->eh_cmd_q
254 - set SHOST_RECOVERY
255 - shost->host_failed++
256 LOCKING: shost->host_lock
257
258 2. EH starts
259 ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
260 is cleared.
261 LOCKING: shost->host_lock (not strictly necessary, just for
262 consistency)
263
264 3. scmd recovered
265 ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
266 - shost->host_failed--
267 - clear scmd->eh_eflags
268 - scsi_setup_cmd_retry()
269 - move from local eh_work_q to local eh_done_q
270 LOCKING: none
271
272 4. EH completes
273 ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
274 layer of failure.
275 - scmd is removed from eh_done_q and scmd->eh_entry is cleared
276 - if retry is necessary, scmd is requeued using
277 scsi_queue_insert()
278 - otherwise, scsi_finish_command() is invoked for scmd
279 LOCKING: queue or finish function performs appropriate locking
280
281
282[2-1-3] Flow of control
283
284 EH through fine-grained callbacks start from scsi_unjam_host().
285
286<<scsi_unjam_host>>
287
288 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
289 eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
290 cleared by this action.
291
292 2. Invoke scsi_eh_get_sense.
293
294 <<scsi_eh_get_sense>>
295
296 This action is taken for each error-completed
297 (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most
298 SCSI transports/LLDDs automatically acquire sense data on
299 command failures (autosense). Autosense is recommended for
300 performance reasons and as sense information could get out of
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300301 sync between occurrence of CHECK CONDITION and this action.
Tejun Heo70c83e12005-09-11 09:37:19 +0900302
303 Note that if autosense is not supported, scmd->sense_buffer
304 contains invalid sense data when error-completing the scmd
305 with scsi_done(). scsi_decide_disposition() always returns
306 FAILED in such cases thus invoking SCSI EH. When the scmd
307 reaches here, sense data is acquired and
308 scsi_decide_disposition() is called again.
309
310 1. Invoke scsi_request_sense() which issues REQUEST_SENSE
311 command. If fails, no action. Note that taking no action
312 causes higher-severity recovery to be taken for the scmd.
313
314 2. Invoke scsi_decide_disposition() on the scmd
315
316 - SUCCESS
317 scmd->retries is set to scmd->allowed preventing
318 scsi_eh_flush_done_q() from retrying the scmd and
319 scsi_eh_finish_cmd() is invoked.
320
321 - NEEDS_RETRY
322 scsi_eh_finish_cmd() invoked
323
324 - otherwise
325 No action.
326
327 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
328
329 <<scsi_eh_abort_cmds>>
330
Hannes Reinecke6ad55502013-11-11 13:44:57 +0100331 This action is taken for each timed out command when
332 no_async_abort is enabled in the host template.
Tejun Heo70c83e12005-09-11 09:37:19 +0900333 hostt->eh_abort_handler() is invoked for each scmd. The
334 handler returns SUCCESS if it has succeeded to make LLDD and
335 all related hardware forget about the scmd.
336
337 If a timedout scmd is successfully aborted and the sdev is
338 either offline or ready, scsi_eh_finish_cmd() is invoked for
339 the scmd. Otherwise, the scmd is left in eh_work_q for
340 higher-severity actions.
341
342 Note that both offline and ready status mean that the sdev is
343 ready to process new scmds, where processing also implies
344 immediate failing; thus, if a sdev is in one of the two
345 states, no further recovery action is needed.
346
347 Device readiness is tested using scsi_eh_tur() which issues
348 TEST_UNIT_READY command. Note that the scmd must have been
349 aborted successfully before reusing it for TEST_UNIT_READY.
350
351 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
352
353 <<scsi_eh_ready_devs>>
354
355 This function takes four increasingly more severe measures to
356 make failed sdevs ready for new commands.
357
358 1. Invoke scsi_eh_stu()
359
360 <<scsi_eh_stu>>
361
362 For each sdev which has failed scmds with valid sense data
363 of which scsi_check_sense()'s verdict is FAILED,
364 START_STOP_UNIT command is issued w/ start=1. Note that
365 as we explicitly choose error-completed scmds, it is known
366 that lower layers have forgotten about the scmd and we can
367 reuse it for STU.
368
369 If STU succeeds and the sdev is either offline or ready,
370 all failed scmds on the sdev are EH-finished with
371 scsi_eh_finish_cmd().
372
373 *NOTE* If hostt->eh_abort_handler() isn't implemented or
374 failed, we may still have timed out scmds at this point
375 and STU doesn't make lower layers forget about those
376 scmds. Yet, this function EH-finish all scmds on the sdev
377 if STU succeeds leaving lower layers in an inconsistent
378 state. It seems that STU action should be taken only when
379 a sdev has no timed out scmd.
380
381 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
382
383 <<scsi_eh_bus_device_reset>>
384
385 This action is very similar to scsi_eh_stu() except that,
386 instead of issuing STU, hostt->eh_device_reset_handler()
387 is used. Also, as we're not issuing SCSI commands and
388 resetting clears all scmds on the sdev, there is no need
389 to choose error-completed scmds.
390
391 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
392
393 <<scsi_eh_bus_reset>>
394
395 hostt->eh_bus_reset_handler() is invoked for each channel
396 with failed scmds. If bus reset succeeds, all failed
397 scmds on all ready or offline sdevs on the channel are
398 EH-finished.
399
400 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
401
402 <<scsi_eh_host_reset>>
403
404 This is the last resort. hostt->eh_host_reset_handler()
405 is invoked. If host reset succeeds, all failed scmds on
406 all ready or offline sdevs on the host are EH-finished.
407
408 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
409
410 <<scsi_eh_offline_sdevs>>
411
412 Take all sdevs which still have unrecovered scmds offline
413 and EH-finish the scmds.
414
415 5. Invoke scsi_eh_flush_done_q().
416
417 <<scsi_eh_flush_done_q>>
418
419 At this point all scmds are recovered (or given up) and
420 put on eh_done_q by scsi_eh_finish_cmd(). This function
421 flushes eh_done_q by either retrying or notifying upper
422 layer of failure of the scmds.
423
424
Christoph Hellwig9227c332006-04-01 19:21:04 +0200425[2-2] EH through transportt->eh_strategy_handler()
Tejun Heo70c83e12005-09-11 09:37:19 +0900426
Christoph Hellwig9227c332006-04-01 19:21:04 +0200427 transportt->eh_strategy_handler() is invoked in the place of
Tejun Heo70c83e12005-09-11 09:37:19 +0900428scsi_unjam_host() and it is responsible for whole recovery process.
429On completion, the handler should have made lower layers forget about
430all failed scmds and either ready for new commands or offline. Also,
Finn Thain542cb452014-10-03 11:42:17 +1000431it should perform SCSI EH maintenance chores to maintain integrity of
Tejun Heo70c83e12005-09-11 09:37:19 +0900432SCSI midlayer. IOW, of the steps described in [2-1-2], all steps
433except for #1 must be implemented by eh_strategy_handler().
434
435
Christoph Hellwig9227c332006-04-01 19:21:04 +0200436[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +0900437
438 The following conditions are true on entry to the handler.
439
440 - Each failed scmd's eh_flags field is set appropriately.
441
442 - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
443
444 - SHOST_RECOVERY is set.
445
446 - shost->host_failed == shost->host_busy
447
448
Christoph Hellwig9227c332006-04-01 19:21:04 +0200449[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
Tejun Heo70c83e12005-09-11 09:37:19 +0900450
451 The following conditions must be true on exit from the handler.
452
453 - shost->host_failed is zero.
454
455 - Each scmd's eh_eflags field is cleared.
456
457 - Each scmd is in such a state that scsi_setup_cmd_retry() on the
458 scmd doesn't make any difference.
459
460 - shost->eh_cmd_q is cleared.
461
462 - Each scmd->eh_entry is cleared.
463
464 - Either scsi_queue_insert() or scsi_finish_command() is called on
465 each scmd. Note that the handler is free to use scmd->retries and
466 ->allowed to limit the number of retries.
467
468
469[2-2-3] Things to consider
470
471 - Know that timed out scmds are still active on lower layers. Make
472 lower layers forget about them before doing anything else with
473 those scmds.
474
475 - For consistency, when accessing/modifying shost data structure,
476 grab shost->host_lock.
477
478 - On completion, each failed sdev must have forgotten about all
479 active scmds.
480
481 - On completion, each failed sdev must be ready for new commands or
482 offline.
483
484
485--
486Tejun Heo
487htejun@gmail.com
48811th September 2005