blob: fdcc49fad8e11d3bb1c00b6033da3a35f33516c6 [file] [log] [blame]
Alan Coxda9bb1d2006-01-18 17:44:13 -08001
2
3EDAC - Error Detection And Correction
4
Doug Thompson87f24c32007-07-19 01:50:34 -07005Written by Doug Thompson <dougthompson@xmission.com>
Alan Coxda9bb1d2006-01-18 17:44:13 -080067 Dec 2005
Doug Thompson87f24c32007-07-19 01:50:34 -0700717 Jul 2007 Updated
Alan Coxda9bb1d2006-01-18 17:44:13 -08008
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -03009(c) Mauro Carvalho Chehab <mchehab@redhat.com>
1005 Aug 2009 Nehalem interface
Alan Coxda9bb1d2006-01-18 17:44:13 -080011
Doug Thompson87f24c32007-07-19 01:50:34 -070012EDAC is maintained and written by:
Alan Coxda9bb1d2006-01-18 17:44:13 -080013
Doug Thompson87f24c32007-07-19 01:50:34 -070014 Doug Thompson, Dave Jiang, Dave Peterson et al,
15 original author: Thayne Harbaugh,
16
17Contact:
18 website: bluesmoke.sourceforge.net
19 mailing list: bluesmoke-devel@lists.sourceforge.net
20
21"bluesmoke" was the name for this device driver when it was "out-of-tree"
22and maintained at sourceforge.net. When it was pushed into 2.6.16 for the
23first time, it was renamed to 'EDAC'.
24
25The bluesmoke project at sourceforge.net is now utilized as a 'staging area'
26for EDAC development, before it is sent upstream to kernel.org
27
Matt LaPlante19f59462009-04-27 15:06:31 +020028At the bluesmoke/EDAC project site is a series of quilt patches against
29recent kernels, stored in a SVN repository. For easier downloading, there
Doug Thompson87f24c32007-07-19 01:50:34 -070030is also a tarball snapshot available.
Alan Coxda9bb1d2006-01-18 17:44:13 -080031
32============================================================================
33EDAC PURPOSE
34
35The 'edac' kernel module goal is to detect and report errors that occur
Doug Thompson87f24c32007-07-19 01:50:34 -070036within the computer system running under linux.
37
38MEMORY
39
40In the initial release, memory Correctable Errors (CE) and Uncorrectable
41Errors (UE) are the primary errors being harvested. These types of errors
42are harvested by the 'edac_mc' class of device.
Alan Coxda9bb1d2006-01-18 17:44:13 -080043
44Detecting CE events, then harvesting those events and reporting them,
45CAN be a predictor of future UE events. With CE events, the system can
Dave Petersonf3479812006-03-26 01:38:53 -080046continue to operate, but with less safety. Preventive maintenance and
Alan Coxda9bb1d2006-01-18 17:44:13 -080047proactive part replacement of memory DIMMs exhibiting CEs can reduce
48the likelihood of the dreaded UE events and system 'panics'.
49
Doug Thompson87f24c32007-07-19 01:50:34 -070050NON-MEMORY
51
52A new feature for EDAC, the edac_device class of device, was added in
53the 2.6.23 version of the kernel.
54
55This new device type allows for non-memory type of ECC hardware detectors
56to have their states harvested and presented to userspace via the sysfs
57interface.
58
59Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA
60engines, fabric switches, main data path switches, interconnections,
61and various other hardware data paths. If the hardware reports it, then
62a edac_device device probably can be constructed to harvest and present
63that to userspace.
64
65
66PCI BUS SCANNING
Alan Coxda9bb1d2006-01-18 17:44:13 -080067
68In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
69in order to determine if errors are occurring on data transfers.
Doug Thompson87f24c32007-07-19 01:50:34 -070070
Alan Coxda9bb1d2006-01-18 17:44:13 -080071The presence of PCI Parity errors must be examined with a grain of salt.
Dave Petersonf3479812006-03-26 01:38:53 -080072There are several add-in adapters that do NOT follow the PCI specification
Alan Coxda9bb1d2006-01-18 17:44:13 -080073with regards to Parity generation and reporting. The specification says
74the vendor should tie the parity status bits to 0 if they do not intend
75to generate parity. Some vendors do not do this, and thus the parity bit
76can "float" giving false positives.
77
Matt LaPlante19f59462009-04-27 15:06:31 +020078In the kernel there is a PCI device attribute located in sysfs that is
Doug Thompson87f24c32007-07-19 01:50:34 -070079checked by the EDAC PCI scanning code. If that attribute is set,
Matt LaPlante19f59462009-04-27 15:06:31 +020080PCI parity/error scanning is skipped for that device. The attribute
Doug Thompson87f24c32007-07-19 01:50:34 -070081is:
82
83 broken_parity_status
84
Davidlohr Buesoe281d752009-11-04 15:30:34 -030085as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for
Doug Thompson87f24c32007-07-19 01:50:34 -070086PCI devices.
87
88FUTURE HARDWARE SCANNING
Alan Coxda9bb1d2006-01-18 17:44:13 -080089
Doug Thompson49c0dab72006-07-10 04:45:19 -070090EDAC will have future error detectors that will be integrated with
91EDAC or added to it, in the following list:
Alan Coxda9bb1d2006-01-18 17:44:13 -080092
93 MCE Machine Check Exception
94 MCA Machine Check Architecture
95 NMI NMI notification of ECC errors
96 MSRs Machine Specific Register error cases
97 and other mechanisms.
98
99These errors are usually bus errors, ECC errors, thermal throttling
100and the like.
101
102
103============================================================================
104EDAC VERSIONING
105
Doug Thompson87f24c32007-07-19 01:50:34 -0700106EDAC is composed of a "core" module (edac_core.ko) and several Memory
Alan Coxda9bb1d2006-01-18 17:44:13 -0800107Controller (MC) driver modules. On a given system, the CORE
108is loaded and one MC driver will be loaded. Both the CORE and
Doug Thompson87f24c32007-07-19 01:50:34 -0700109the MC driver (or edac_device driver) have individual versions that reflect
110current release level of their respective modules.
111
112Thus, to "report" on what version a system is running, one must report both
113the CORE's and the MC driver's versions.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800114
115
116LOADING
117
118If 'edac' was statically linked with the kernel then no loading is
119necessary. If 'edac' was built as modules then simply modprobe the
120'edac' pieces that you need. You should be able to modprobe
121hardware-specific modules and have the dependencies load the necessary core
122modules.
123
124Example:
125
126$> modprobe amd76x_edac
127
128loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
129core module.
130
131
132============================================================================
133EDAC sysfs INTERFACE
134
135EDAC presents a 'sysfs' interface for control, reporting and attribute
136reporting purposes.
137
Doug Thompson87f24c32007-07-19 01:50:34 -0700138EDAC lives in the /sys/devices/system/edac directory.
139
140Within this directory there currently reside 2 'edac' components:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800141
142 mc memory controller(s) system
Doug Thompson49c0dab72006-07-10 04:45:19 -0700143 pci PCI control and status system
Alan Coxda9bb1d2006-01-18 17:44:13 -0800144
145
146============================================================================
147Memory Controller (mc) Model
148
149First a background on the memory controller's model abstracted in EDAC.
Doug Thompson49c0dab72006-07-10 04:45:19 -0700150Each 'mc' device controls a set of DIMM memory modules. These modules are
Dave Petersonf3479812006-03-26 01:38:53 -0800151laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
Doug Thompson49c0dab72006-07-10 04:45:19 -0700152be multiple csrows and multiple channels.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800153
154Memory controllers allow for several csrows, with 8 csrows being a typical value.
155Yet, the actual number of csrows depends on the electrical "loading"
156of a given motherboard, memory controller and DIMM characteristics.
157
158Dual channels allows for 128 bit data transfers to the CPU from memory.
Doug Thompson49c0dab72006-07-10 04:45:19 -0700159Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
160(FB-DIMMs). The following example will assume 2 channels:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800161
162
163 Channel 0 Channel 1
164 ===================================
165 csrow0 | DIMM_A0 | DIMM_B0 |
166 csrow1 | DIMM_A0 | DIMM_B0 |
167 ===================================
168
169 ===================================
170 csrow2 | DIMM_A1 | DIMM_B1 |
171 csrow3 | DIMM_A1 | DIMM_B1 |
172 ===================================
173
174In the above example table there are 4 physical slots on the motherboard
175for memory DIMMs:
176
177 DIMM_A0
178 DIMM_B0
179 DIMM_A1
180 DIMM_B1
181
182Labels for these slots are usually silk screened on the motherboard. Slots
Dave Petersonf3479812006-03-26 01:38:53 -0800183labeled 'A' are channel 0 in this example. Slots labeled 'B'
Alan Coxda9bb1d2006-01-18 17:44:13 -0800184are channel 1. Notice that there are two csrows possible on a
185physical DIMM. These csrows are allocated their csrow assignment
186based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
187is placed in each Channel, the csrows cross both DIMMs.
188
189Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
190Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
191will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
Dave Petersonf3479812006-03-26 01:38:53 -0800192when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
Alan Coxda9bb1d2006-01-18 17:44:13 -0800193csrow1 will be populated. The pattern repeats itself for csrow2 and
194csrow3.
195
196The representation of the above is reflected in the directory tree
197in EDAC's sysfs interface. Starting in directory
198/sys/devices/system/edac/mc each memory controller will be represented
Raoul Bhatia5989f112010-11-25 17:32:47 +0100199by its own 'mcX' directory, where 'X' is the index of the MC.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800200
201
202 ..../edac/mc/
203 |
204 |->mc0
205 |->mc1
206 |->mc2
207 ....
208
209Under each 'mcX' directory each 'csrowX' is again represented by a
Raoul Bhatia5989f112010-11-25 17:32:47 +0100210'csrowX', where 'X' is the csrow index:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800211
212
213 .../mc/mc0/
214 |
215 |->csrow0
216 |->csrow2
217 |->csrow3
218 ....
219
220Notice that there is no csrow1, which indicates that csrow0 is
221composed of a single ranked DIMMs. This should also apply in both
222Channels, in order to have dual-channel mode be operational. Since
223both csrow2 and csrow3 are populated, this indicates a dual ranked
224set of DIMMs for channels 0 and 1.
225
226
Arthur Jones327dafb2008-07-25 01:49:10 -0700227Within each of the 'mcX' and 'csrowX' directories are several
Alan Coxda9bb1d2006-01-18 17:44:13 -0800228EDAC control and attribute files.
229
Alan Coxda9bb1d2006-01-18 17:44:13 -0800230============================================================================
231'mcX' DIRECTORIES
232
233
234In 'mcX' directories are EDAC control and attribute files for
Raoul Bhatia5989f112010-11-25 17:32:47 +0100235this 'X' instance of the memory controllers:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800236
237
238Counter reset control file:
239
240 'reset_counters'
241
242 This write-only control file will zero all the statistical counters
243 for UE and CE errors. Zeroing the counters will also reset the timer
244 indicating how long since the last counter zero. This is useful
245 for computing errors/time. Since the counters are always reset at
246 driver initialization time, no module/kernel parameter is available.
247
248 RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
249
250 This resets the counters on memory controller 0
251
252
253Seconds since last counter reset control file:
254
255 'seconds_since_reset'
256
257 This attribute file displays how many seconds have elapsed since the
258 last counter reset. This can be used with the error counters to
259 measure error rates.
260
261
262
Alan Coxda9bb1d2006-01-18 17:44:13 -0800263Memory Controller name attribute file:
264
265 'mc_name'
266
267 This attribute file displays the type of memory controller
268 that is being utilized.
269
270
Alan Coxda9bb1d2006-01-18 17:44:13 -0800271Total memory managed by this memory controller attribute file:
272
273 'size_mb'
274
275 This attribute file displays, in count of megabytes, of memory
276 that this instance of memory controller manages.
277
278
279Total Uncorrectable Errors count attribute file:
280
281 'ue_count'
282
283 This attribute file displays the total count of uncorrectable
284 errors that have occurred on this memory controller. If panic_on_ue
285 is set this counter will not have a chance to increment,
286 since EDAC will panic the system.
287
288
289Total UE count that had no information attribute fileY:
290
291 'ue_noinfo_count'
292
Davidlohr Buesoe281d752009-11-04 15:30:34 -0300293 This attribute file displays the number of UEs that have occurred
294 with no information as to which DIMM slot is having errors.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800295
296
297Total Correctable Errors count attribute file:
298
299 'ce_count'
300
301 This attribute file displays the total count of correctable
302 errors that have occurred on this memory controller. This
303 count is very important to examine. CEs provide early
304 indications that a DIMM is beginning to fail. This count
305 field should be monitored for non-zero values and report
306 such information to the system administrator.
307
308
309Total Correctable Errors count attribute file:
310
311 'ce_noinfo_count'
312
313 This attribute file displays the number of CEs that
Sylvestre Ledruf65e51d2011-04-04 15:04:46 -0700314 have occurred wherewith no information as to which DIMM slot
Alan Coxda9bb1d2006-01-18 17:44:13 -0800315 is having errors. Memory is handicapped, but operational,
316 yet no information is available to indicate which slot
317 the failing memory is in. This count field should be also
318 be monitored for non-zero values.
319
320Device Symlink:
321
322 'device'
323
Frithiof Jensen4f423dd2007-02-12 00:53:07 -0800324 Symlink to the memory controller device.
325
326Sdram memory scrubbing rate:
327
328 'sdram_scrub_rate'
329
330 Read/Write attribute file that controls memory scrubbing. The scrubbing
Matt LaPlanted9195882008-07-25 19:45:33 -0700331 rate is set by writing a minimum bandwidth in bytes/sec to the attribute
Frithiof Jensen4f423dd2007-02-12 00:53:07 -0800332 file. The rate will be translated to an internal value that gives at
333 least the specified rate.
334
335 Reading the file will return the actual scrubbing rate employed.
336
Borislav Petkov5e8e19b2011-09-21 14:10:43 +0200337 If configuration fails or memory scrubbing is not implemented, accessing
338 that attribute will fail.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800339
340
341
342============================================================================
343'csrowX' DIRECTORIES
344
345In the 'csrowX' directories are EDAC control and attribute files for
Raoul Bhatia5989f112010-11-25 17:32:47 +0100346this 'X' instance of csrow:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800347
348
349Total Uncorrectable Errors count attribute file:
350
351 'ue_count'
352
353 This attribute file displays the total count of uncorrectable
354 errors that have occurred on this csrow. If panic_on_ue is set
355 this counter will not have a chance to increment, since EDAC
356 will panic the system.
357
358
359Total Correctable Errors count attribute file:
360
361 'ce_count'
362
363 This attribute file displays the total count of correctable
364 errors that have occurred on this csrow. This
365 count is very important to examine. CEs provide early
366 indications that a DIMM is beginning to fail. This count
367 field should be monitored for non-zero values and report
368 such information to the system administrator.
369
370
371Total memory managed by this csrow attribute file:
372
373 'size_mb'
374
375 This attribute file displays, in count of megabytes, of memory
Dave Petersonf3479812006-03-26 01:38:53 -0800376 that this csrow contains.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800377
378
379Memory Type attribute file:
380
381 'mem_type'
382
383 This attribute file will display what type of memory is currently
384 on this csrow. Normally, either buffered or unbuffered memory.
Doug Thompson49c0dab72006-07-10 04:45:19 -0700385 Examples:
386 Registered-DDR
387 Unbuffered-DDR
Alan Coxda9bb1d2006-01-18 17:44:13 -0800388
389
390EDAC Mode of operation attribute file:
391
392 'edac_mode'
393
394 This attribute file will display what type of Error detection
395 and correction is being utilized.
396
397
398Device type attribute file:
399
400 'dev_type'
401
Doug Thompson49c0dab72006-07-10 04:45:19 -0700402 This attribute file will display what type of DRAM device is
403 being utilized on this DIMM.
404 Examples:
405 x1
406 x2
407 x4
408 x8
Alan Coxda9bb1d2006-01-18 17:44:13 -0800409
410
411Channel 0 CE Count attribute file:
412
413 'ch0_ce_count'
414
415 This attribute file will display the count of CEs on this
416 DIMM located in channel 0.
417
418
419Channel 0 UE Count attribute file:
420
421 'ch0_ue_count'
422
423 This attribute file will display the count of UEs on this
424 DIMM located in channel 0.
425
426
427Channel 0 DIMM Label control file:
428
429 'ch0_dimm_label'
430
431 This control file allows this DIMM to have a label assigned
432 to it. With this label in the module, when errors occur
433 the output can provide the DIMM label in the system log.
434 This becomes vital for panic events to isolate the
435 cause of the UE event.
436
437 DIMM Labels must be assigned after booting, with information
438 that correctly identifies the physical slot with its
439 silk screen label. This information is currently very
440 motherboard specific and determination of this information
441 must occur in userland at this time.
442
443
444Channel 1 CE Count attribute file:
445
446 'ch1_ce_count'
447
448 This attribute file will display the count of CEs on this
449 DIMM located in channel 1.
450
451
452Channel 1 UE Count attribute file:
453
454 'ch1_ue_count'
455
456 This attribute file will display the count of UEs on this
457 DIMM located in channel 0.
458
459
460Channel 1 DIMM Label control file:
461
462 'ch1_dimm_label'
463
464 This control file allows this DIMM to have a label assigned
465 to it. With this label in the module, when errors occur
466 the output can provide the DIMM label in the system log.
467 This becomes vital for panic events to isolate the
468 cause of the UE event.
469
470 DIMM Labels must be assigned after booting, with information
471 that correctly identifies the physical slot with its
472 silk screen label. This information is currently very
473 motherboard specific and determination of this information
474 must occur in userland at this time.
475
Alan Coxda9bb1d2006-01-18 17:44:13 -0800476============================================================================
477SYSTEM LOGGING
478
479If logging for UEs and CEs are enabled then system logs will have
480error notices indicating errors that have been detected:
481
Doug Thompson49c0dab72006-07-10 04:45:19 -0700482EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800483channel 1 "DIMM_B1": amd76x_edac
484
Doug Thompson49c0dab72006-07-10 04:45:19 -0700485EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800486channel 1 "DIMM_B1": amd76x_edac
487
488
489The structure of the message is:
490 the memory controller (MC0)
491 Error type (CE)
492 memory page (0x283)
493 offset in the page (0xce0)
494 the byte granularity (grain 8)
495 or resolution of the error
496 the error syndrome (0xb741)
497 memory row (row 0)
498 memory channel (channel 1)
499 DIMM label, if set prior (DIMM B1
500 and then an optional, driver-specific message that may
501 have additional information.
502
503Both UEs and CEs with no info will lack all but memory controller,
504error type, a notice of "no info" and then an optional,
505driver-specific error message.
506
507
Alan Coxda9bb1d2006-01-18 17:44:13 -0800508============================================================================
509PCI Bus Parity Detection
510
511
512On Header Type 00 devices the primary status is looked at
513for any parity error regardless of whether Parity is enabled on the
514device. (The spec indicates parity is generated in some cases).
515On Header Type 01 bridges, the secondary status register is also
Dave Petersonf3479812006-03-26 01:38:53 -0800516looked at to see if parity occurred on the bus on the other side of
Alan Coxda9bb1d2006-01-18 17:44:13 -0800517the bridge.
518
519
520SYSFS CONFIGURATION
521
522Under /sys/devices/system/edac/pci are control and attribute files as follows:
523
524
525Enable/Disable PCI Parity checking control file:
526
527 'check_pci_parity'
528
529
530 This control file enables or disables the PCI Bus Parity scanning
531 operation. Writing a 1 to this file enables the scanning. Writing
532 a 0 to this file disables the scanning.
533
534 Enable:
535 echo "1" >/sys/devices/system/edac/pci/check_pci_parity
536
537 Disable:
538 echo "0" >/sys/devices/system/edac/pci/check_pci_parity
539
540
Arthur Jones327dafb2008-07-25 01:49:10 -0700541Parity Count:
542
543 'pci_parity_count'
544
545 This attribute file will display the number of parity errors that
546 have been detected.
547
548
549============================================================================
550MODULE PARAMETERS
551
552Panic on UE control file:
553
554 'edac_mc_panic_on_ue'
555
556 An uncorrectable error will cause a machine panic. This is usually
557 desirable. It is a bad idea to continue when an uncorrectable error
558 occurs - it is indeterminate what was uncorrected and the operating
559 system context might be so mangled that continuing will lead to further
560 corruption. If the kernel has MCE configured, then EDAC will never
561 notice the UE.
562
563 LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1]
564
565 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
566
567
568Log UE control file:
569
570 'edac_mc_log_ue'
571
572 Generate kernel messages describing uncorrectable errors. These errors
573 are reported through the system message log system. UE statistics
574 will be accumulated even when UE logging is disabled.
575
576 LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1]
577
578 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
579
580
581Log CE control file:
582
583 'edac_mc_log_ce'
584
585 Generate kernel messages describing correctable errors. These
586 errors are reported through the system message log system.
587 CE statistics will be accumulated even when CE logging is disabled.
588
589 LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1]
590
591 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
592
593
594Polling period control file:
595
596 'edac_mc_poll_msec'
597
598 The time period, in milliseconds, for polling for error information.
599 Too small a value wastes resources. Too large a value might delay
600 necessary handling of errors and might loose valuable information for
601 locating the error. 1000 milliseconds (once each second) is the current
602 default. Systems which require all the bandwidth they can get, may
603 increase this.
604
605 LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1]
606
607 RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
608
Alan Coxda9bb1d2006-01-18 17:44:13 -0800609
610Panic on PCI PARITY Error:
611
612 'panic_on_pci_parity'
613
614
Dave Petersonf3479812006-03-26 01:38:53 -0800615 This control files enables or disables panicking when a parity
Alan Coxda9bb1d2006-01-18 17:44:13 -0800616 error has been detected.
617
618
Arthur Jones327dafb2008-07-25 01:49:10 -0700619 module/kernel parameter: edac_panic_on_pci_pe=[0|1]
Alan Coxda9bb1d2006-01-18 17:44:13 -0800620
621 Enable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700622 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800623
624 Disable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700625 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800626
627
628
Alan Coxda9bb1d2006-01-18 17:44:13 -0800629=======================================================================
Doug Thompson87f24c32007-07-19 01:50:34 -0700630
631
632EDAC_DEVICE type of device
633
634In the header file, edac_core.h, there is a series of edac_device structures
635and APIs for the EDAC_DEVICE.
636
637User space access to an edac_device is through the sysfs interface.
638
639At the location /sys/devices/system/edac (sysfs) new edac_device devices will
640appear.
641
642There is a three level tree beneath the above 'edac' directory. For example,
643the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
644installs itself as:
645
646 /sys/devices/systm/edac/test-instance
647
648in this directory are various controls, a symlink and one or more 'instance'
649directorys.
650
651The standard default controls are:
652
653 log_ce boolean to log CE events
654 log_ue boolean to log UE events
655 panic_on_ue boolean to 'panic' the system if an UE is encountered
656 (default off, can be set true via startup script)
657 poll_msec time period between POLL cycles for events
658
659The test_device_edac device adds at least one of its own custom control:
660
661 test_bits which in the current test driver does nothing but
662 show how it is installed. A ported driver can
663 add one or more such controls and/or attributes
664 for specific uses.
665 One out-of-tree driver uses controls here to allow
666 for ERROR INJECTION operations to hardware
667 injection registers
668
669The symlink points to the 'struct dev' that is registered for this edac_device.
670
671INSTANCES
672
673One or more instance directories are present. For the 'test_device_edac' case:
674
675 test-instance0
676
677
678In this directory there are two default counter attributes, which are totals of
679counter in deeper subdirectories.
680
681 ce_count total of CE events of subdirectories
682 ue_count total of UE events of subdirectories
683
684BLOCKS
685
686At the lowest directory level is the 'block' directory. There can be 0, 1
687or more blocks specified in each instance.
688
689 test-block0
690
691
692In this directory the default attributes are:
693
694 ce_count which is counter of CE events for this 'block'
695 of hardware being monitored
696 ue_count which is counter of UE events for this 'block'
697 of hardware being monitored
698
699
700The 'test_device_edac' device adds 4 attributes and 1 control:
701
702 test-block-bits-0 for every POLL cycle this counter
703 is incremented
704 test-block-bits-1 every 10 cycles, this counter is bumped once,
705 and test-block-bits-0 is set to 0
706 test-block-bits-2 every 100 cycles, this counter is bumped once,
707 and test-block-bits-1 is set to 0
708 test-block-bits-3 every 1000 cycles, this counter is bumped once,
709 and test-block-bits-2 is set to 0
710
711
712 reset-counters writing ANY thing to this control will
713 reset all the above counters.
714
715
716Use of the 'test_device_edac' driver should any others to create their own
717unique drivers for their hardware systems.
718
719The 'test_device_edac' sample driver is located at the
720bluesmoke.sourceforge.net project site for EDAC.
721
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300722=======================================================================
723NEHALEM USAGE OF EDAC APIs
724
725This chapter documents some EXPERIMENTAL mappings for EDAC API to handle
726Nehalem EDAC driver. They will likely be changed on future versions
727of the driver.
728
729Due to the way Nehalem exports Memory Controller data, some adjustments
730were done at i7core_edac driver. This chapter will cover those differences
731
7321) On Nehalem, there are one Memory Controller per Quick Patch Interconnect
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300733 (QPI). At the driver, the term "socket" means one QPI. This is
734 associated with a physical CPU socket.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300735
736 Each MC have 3 physical read channels, 3 physical write channels and
737 3 logic channels. The driver currenty sees it as just 3 channels.
738 Each channel can have up to 3 DIMMs.
739
740 The minimum known unity is DIMMs. There are no information about csrows.
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300741 As EDAC API maps the minimum unity is csrows, the driver sequencially
742 maps channel/dimm into different csrows.
743
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300744 For example, supposing the following layout:
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300745 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
746 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
747 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
748 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
749 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
750 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
751 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
752 The driver will map it as:
753 csrow0: channel 0, dimm0
754 csrow1: channel 0, dimm1
755 csrow2: channel 1, dimm0
756 csrow3: channel 2, dimm0
757
758exports one
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300759 DIMM per csrow.
760
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300761 Each QPI is exported as a different memory controller.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300762
7632) Nehalem MC has the hability to generate errors. The driver implements this
764 functionality via some error injection nodes:
765
766 For injecting a memory error, there are some sysfs nodes, under
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300767 /sys/devices/system/edac/mc/mc?/:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300768
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300769 inject_addrmatch/*:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300770 Controls the error injection mask register. It is possible to specify
771 several characteristics of the address to match an error code:
772 dimm = the affected dimm. Numbers are relative to a channel;
773 rank = the memory rank;
774 channel = the channel that will generate an error;
775 bank = the affected bank;
776 page = the page address;
777 column (or col) = the address column.
778 each of the above values can be set to "any" to match any valid value.
779
780 At driver init, all values are set to any.
781
782 For example, to generate an error at rank 1 of dimm 2, for any channel,
783 any bank, any page, any column:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300784 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
785 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300786
787 To return to the default behaviour of matching any, you can do:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300788 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
789 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300790
791 inject_eccmask:
792 specifies what bits will have troubles,
793
794 inject_section:
795 specifies what ECC cache section will get the error:
796 3 for both
797 2 for the highest
798 1 for the lowest
799
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300800 inject_type:
801 specifies the type of error, being a combination of the following bits:
802 bit 0 - repeat
803 bit 1 - ecc
804 bit 2 - parity
805
806 inject_enable starts the error generation when something different
807 than 0 is written.
808
809 All inject vars can be read. root permission is needed for write.
810
811 Datasheet states that the error will only be generated after a write on an
812 address that matches inject_addrmatch. It seems, however, that reading will
813 also produce an error.
814
815 For example, the following code will generate an error for any write access
816 at socket 0, on any DIMM/address on channel 2:
817
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300818 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300819 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
820 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
821 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300822 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
823 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
824
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300825 For socket 1, it is needed to replace "mc0" by "mc1" at the above
826 commands.
827
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300828 The generated error message will look like:
829
830 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
831
8323) Nehalem specific Corrected Error memory counters
833
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300834 Nehalem have some registers to count memory errors. The driver uses those
835 registers to report Corrected Errors on devices with Registered Dimms.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300836
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300837 However, those counters don't work with Unregistered Dimms. As the chipset
838 offers some counters that also work with UDIMMS (but with a worse level of
839 granularity than the default ones), the driver exposes those registers for
840 UDIMM memories.
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300841
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300842 They can be read by looking at the contents of all_channel_counts/
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300843
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300844 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
845 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
846 0
847 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
848 0
849 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
850 0
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300851
852 What happens here is that errors on different csrows, but at the same
853 dimm number will increment the same counter.
854 So, in this memory mapping:
855 csrow0: channel 0, dimm0
856 csrow1: channel 0, dimm1
857 csrow2: channel 1, dimm0
858 csrow3: channel 2, dimm0
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300859 The hardware will increment udimm0 for an error at the first dimm at either
860 csrow0, csrow2 or csrow3;
861 The hardware will increment udimm1 for an error at the second dimm at either
862 csrow0, csrow2 or csrow3;
863 The hardware will increment udimm2 for an error at the third dimm at either
864 csrow0, csrow2 or csrow3;
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300865
8664) Standard error counters
867
868 The standard error counters are generated when an mcelog error is received
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300869 by the driver. Since, with udimm, this is counted by software, it is
870 possible that some errors could be lost. With rdimm's, they displays the
871 contents of the registers