blob: 80841a2d640cf5aeca8e16db4ebba5d224c5a008 [file] [log] [blame]
Alan Coxda9bb1d2006-01-18 17:44:13 -08001EDAC - Error Detection And Correction
Borislav Petkov043b4312015-06-19 11:47:17 +02002=====================================
Doug Thompson87f24c32007-07-19 01:50:34 -07003
4"bluesmoke" was the name for this device driver when it was "out-of-tree"
5and maintained at sourceforge.net. When it was pushed into 2.6.16 for the
6first time, it was renamed to 'EDAC'.
7
Borislav Petkov043b4312015-06-19 11:47:17 +02008PURPOSE
9-------
Doug Thompson87f24c32007-07-19 01:50:34 -070010
Borislav Petkov043b4312015-06-19 11:47:17 +020011The 'edac' kernel module's goal is to detect and report hardware errors
12that occur within the computer system running under linux.
Doug Thompson87f24c32007-07-19 01:50:34 -070013
14MEMORY
Borislav Petkov043b4312015-06-19 11:47:17 +020015------
Doug Thompson87f24c32007-07-19 01:50:34 -070016
Borislav Petkov043b4312015-06-19 11:47:17 +020017Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
18primary errors being harvested. These types of errors are harvested by
19the 'edac_mc' device.
Alan Coxda9bb1d2006-01-18 17:44:13 -080020
21Detecting CE events, then harvesting those events and reporting them,
Borislav Petkov043b4312015-06-19 11:47:17 +020022*can* but must not necessarily be a predictor of future UE events. With
23CE events only, the system can and will continue to operate as no data
24has been damaged yet.
Alan Coxda9bb1d2006-01-18 17:44:13 -080025
Borislav Petkov043b4312015-06-19 11:47:17 +020026However, preventive maintenance and proactive part replacement of memory
27DIMMs exhibiting CEs can reduce the likelihood of the dreaded UE events
28and system panics.
29
30OTHER HARDWARE ELEMENTS
31-----------------------
Doug Thompson87f24c32007-07-19 01:50:34 -070032
33A new feature for EDAC, the edac_device class of device, was added in
34the 2.6.23 version of the kernel.
35
36This new device type allows for non-memory type of ECC hardware detectors
37to have their states harvested and presented to userspace via the sysfs
38interface.
39
Borislav Petkov043b4312015-06-19 11:47:17 +020040Some architectures have ECC detectors for L1, L2 and L3 caches,
41along with DMA engines, fabric switches, main data path switches,
42interconnections, and various other hardware data paths. If the hardware
43reports it, then a edac_device device probably can be constructed to
44harvest and present that to userspace.
Doug Thompson87f24c32007-07-19 01:50:34 -070045
46
47PCI BUS SCANNING
Borislav Petkov043b4312015-06-19 11:47:17 +020048----------------
Alan Coxda9bb1d2006-01-18 17:44:13 -080049
Borislav Petkov043b4312015-06-19 11:47:17 +020050In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
51in order to determine if errors are occurring during data transfers.
Doug Thompson87f24c32007-07-19 01:50:34 -070052
Alan Coxda9bb1d2006-01-18 17:44:13 -080053The presence of PCI Parity errors must be examined with a grain of salt.
Borislav Petkov043b4312015-06-19 11:47:17 +020054There are several add-in adapters that do *not* follow the PCI specification
Alan Coxda9bb1d2006-01-18 17:44:13 -080055with regards to Parity generation and reporting. The specification says
56the vendor should tie the parity status bits to 0 if they do not intend
57to generate parity. Some vendors do not do this, and thus the parity bit
58can "float" giving false positives.
59
Borislav Petkov043b4312015-06-19 11:47:17 +020060There is a PCI device attribute located in sysfs that is checked by
61the EDAC PCI scanning code. If that attribute is set, PCI parity/error
62scanning is skipped for that device. The attribute is:
Doug Thompson87f24c32007-07-19 01:50:34 -070063
64 broken_parity_status
65
Borislav Petkov043b4312015-06-19 11:47:17 +020066and is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for
Doug Thompson87f24c32007-07-19 01:50:34 -070067PCI devices.
68
Alan Coxda9bb1d2006-01-18 17:44:13 -080069
Borislav Petkov043b4312015-06-19 11:47:17 +020070VERSIONING
71----------
Alan Coxda9bb1d2006-01-18 17:44:13 -080072
Doug Thompson87f24c32007-07-19 01:50:34 -070073EDAC is composed of a "core" module (edac_core.ko) and several Memory
Borislav Petkov043b4312015-06-19 11:47:17 +020074Controller (MC) driver modules. On a given system, the CORE is loaded
75and one MC driver will be loaded. Both the CORE and the MC driver (or
76edac_device driver) have individual versions that reflect current
77release level of their respective modules.
Doug Thompson87f24c32007-07-19 01:50:34 -070078
Borislav Petkov043b4312015-06-19 11:47:17 +020079Thus, to "report" on what version a system is running, one must report
80both the CORE's and the MC driver's versions.
Alan Coxda9bb1d2006-01-18 17:44:13 -080081
82
83LOADING
Borislav Petkov043b4312015-06-19 11:47:17 +020084-------
Alan Coxda9bb1d2006-01-18 17:44:13 -080085
Borislav Petkov043b4312015-06-19 11:47:17 +020086If 'edac' was statically linked with the kernel then no loading
87is necessary. If 'edac' was built as modules then simply modprobe
88the 'edac' pieces that you need. You should be able to modprobe
89hardware-specific modules and have the dependencies load the necessary
90core modules.
Alan Coxda9bb1d2006-01-18 17:44:13 -080091
92Example:
93
94$> modprobe amd76x_edac
95
96loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
97core module.
98
99
Borislav Petkov043b4312015-06-19 11:47:17 +0200100SYSFS INTERFACE
101---------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800102
Borislav Petkov043b4312015-06-19 11:47:17 +0200103EDAC presents a 'sysfs' interface for control and reporting purposes. It
104lives in the /sys/devices/system/edac directory.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800105
Borislav Petkov043b4312015-06-19 11:47:17 +0200106Within this directory there currently reside 2 components:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800107
108 mc memory controller(s) system
Doug Thompson49c0dab72006-07-10 04:45:19 -0700109 pci PCI control and status system
Alan Coxda9bb1d2006-01-18 17:44:13 -0800110
111
Borislav Petkov043b4312015-06-19 11:47:17 +0200112
Alan Coxda9bb1d2006-01-18 17:44:13 -0800113Memory Controller (mc) Model
Borislav Petkov043b4312015-06-19 11:47:17 +0200114----------------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800115
Borislav Petkov043b4312015-06-19 11:47:17 +0200116Each 'mc' device controls a set of DIMM memory modules. These modules
117are laid out in a Chip-Select Row (csrowX) and Channel table (chX).
118There can be multiple csrows and multiple channels.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800119
Borislav Petkov043b4312015-06-19 11:47:17 +0200120Memory controllers allow for several csrows, with 8 csrows being a
121typical value. Yet, the actual number of csrows depends on the layout of
122a given motherboard, memory controller and DIMM characteristics.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800123
Borislav Petkov043b4312015-06-19 11:47:17 +0200124Dual channels allows for 128 bit data transfers to/from the CPU from/to
125memory. Some newer chipsets allow for more than 2 channels, like Fully
126Buffered DIMMs (FB-DIMMs). The following example will assume 2 channels:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800127
128
129 Channel 0 Channel 1
130 ===================================
131 csrow0 | DIMM_A0 | DIMM_B0 |
132 csrow1 | DIMM_A0 | DIMM_B0 |
133 ===================================
134
135 ===================================
136 csrow2 | DIMM_A1 | DIMM_B1 |
137 csrow3 | DIMM_A1 | DIMM_B1 |
138 ===================================
139
140In the above example table there are 4 physical slots on the motherboard
141for memory DIMMs:
142
143 DIMM_A0
144 DIMM_B0
145 DIMM_A1
146 DIMM_B1
147
Borislav Petkov043b4312015-06-19 11:47:17 +0200148Labels for these slots are usually silk-screened on the motherboard.
149Slots labeled 'A' are channel 0 in this example. Slots labeled 'B' are
150channel 1. Notice that there are two csrows possible on a physical DIMM.
151These csrows are allocated their csrow assignment based on the slot into
152which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
153Channel, the csrows cross both DIMMs.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800154
155Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
156Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
157will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
Dave Petersonf3479812006-03-26 01:38:53 -0800158when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
Alan Coxda9bb1d2006-01-18 17:44:13 -0800159csrow1 will be populated. The pattern repeats itself for csrow2 and
160csrow3.
161
Borislav Petkov043b4312015-06-19 11:47:17 +0200162The representation of the above is reflected in the directory
163tree in EDAC's sysfs interface. Starting in directory
Alan Coxda9bb1d2006-01-18 17:44:13 -0800164/sys/devices/system/edac/mc each memory controller will be represented
Raoul Bhatia5989f112010-11-25 17:32:47 +0100165by its own 'mcX' directory, where 'X' is the index of the MC.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800166
167
168 ..../edac/mc/
169 |
170 |->mc0
171 |->mc1
172 |->mc2
173 ....
174
175Under each 'mcX' directory each 'csrowX' is again represented by a
Raoul Bhatia5989f112010-11-25 17:32:47 +0100176'csrowX', where 'X' is the csrow index:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800177
178
179 .../mc/mc0/
180 |
181 |->csrow0
182 |->csrow2
183 |->csrow3
184 ....
185
Borislav Petkov043b4312015-06-19 11:47:17 +0200186Notice that there is no csrow1, which indicates that csrow0 is composed
187of a single ranked DIMMs. This should also apply in both Channels, in
188order to have dual-channel mode be operational. Since both csrow2 and
189csrow3 are populated, this indicates a dual ranked set of DIMMs for
190channels 0 and 1.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800191
192
Borislav Petkov043b4312015-06-19 11:47:17 +0200193Within each of the 'mcX' and 'csrowX' directories are several EDAC
194control and attribute files.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800195
Alan Coxda9bb1d2006-01-18 17:44:13 -0800196
Borislav Petkov043b4312015-06-19 11:47:17 +0200197'mcX' directories
198-----------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800199
200In 'mcX' directories are EDAC control and attribute files for
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300201this 'X' instance of the memory controllers.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800202
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300203For a description of the sysfs API, please see:
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300204 Documentation/ABI/testing/sysfs-devices-edac
Alan Coxda9bb1d2006-01-18 17:44:13 -0800205
206
Alan Coxda9bb1d2006-01-18 17:44:13 -0800207
Borislav Petkov043b4312015-06-19 11:47:17 +0200208'csrowX' directories
209--------------------
210
211When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the csrowX
212directories. As this API doesn't work properly for Rambus, FB-DIMMs and
213modern Intel Memory Controllers, this is being deprecated in favor of
214dimmX directories.
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300215
Alan Coxda9bb1d2006-01-18 17:44:13 -0800216In the 'csrowX' directories are EDAC control and attribute files for
Raoul Bhatia5989f112010-11-25 17:32:47 +0100217this 'X' instance of csrow:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800218
219
220Total Uncorrectable Errors count attribute file:
221
222 'ue_count'
223
224 This attribute file displays the total count of uncorrectable
225 errors that have occurred on this csrow. If panic_on_ue is set
226 this counter will not have a chance to increment, since EDAC
227 will panic the system.
228
229
230Total Correctable Errors count attribute file:
231
232 'ce_count'
233
234 This attribute file displays the total count of correctable
Borislav Petkov043b4312015-06-19 11:47:17 +0200235 errors that have occurred on this csrow. This count is very
236 important to examine. CEs provide early indications that a
237 DIMM is beginning to fail. This count field should be
238 monitored for non-zero values and report such information
239 to the system administrator.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800240
241
242Total memory managed by this csrow attribute file:
243
244 'size_mb'
245
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300246 This attribute file displays, in count of megabytes, the memory
Dave Petersonf3479812006-03-26 01:38:53 -0800247 that this csrow contains.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800248
249
250Memory Type attribute file:
251
252 'mem_type'
253
254 This attribute file will display what type of memory is currently
255 on this csrow. Normally, either buffered or unbuffered memory.
Doug Thompson49c0dab72006-07-10 04:45:19 -0700256 Examples:
257 Registered-DDR
258 Unbuffered-DDR
Alan Coxda9bb1d2006-01-18 17:44:13 -0800259
260
261EDAC Mode of operation attribute file:
262
263 'edac_mode'
264
265 This attribute file will display what type of Error detection
266 and correction is being utilized.
267
268
269Device type attribute file:
270
271 'dev_type'
272
Doug Thompson49c0dab72006-07-10 04:45:19 -0700273 This attribute file will display what type of DRAM device is
274 being utilized on this DIMM.
275 Examples:
276 x1
277 x2
278 x4
279 x8
Alan Coxda9bb1d2006-01-18 17:44:13 -0800280
281
282Channel 0 CE Count attribute file:
283
284 'ch0_ce_count'
285
286 This attribute file will display the count of CEs on this
287 DIMM located in channel 0.
288
289
290Channel 0 UE Count attribute file:
291
292 'ch0_ue_count'
293
294 This attribute file will display the count of UEs on this
295 DIMM located in channel 0.
296
297
298Channel 0 DIMM Label control file:
299
300 'ch0_dimm_label'
301
302 This control file allows this DIMM to have a label assigned
303 to it. With this label in the module, when errors occur
304 the output can provide the DIMM label in the system log.
305 This becomes vital for panic events to isolate the
306 cause of the UE event.
307
308 DIMM Labels must be assigned after booting, with information
309 that correctly identifies the physical slot with its
310 silk screen label. This information is currently very
311 motherboard specific and determination of this information
312 must occur in userland at this time.
313
314
315Channel 1 CE Count attribute file:
316
317 'ch1_ce_count'
318
319 This attribute file will display the count of CEs on this
320 DIMM located in channel 1.
321
322
323Channel 1 UE Count attribute file:
324
325 'ch1_ue_count'
326
327 This attribute file will display the count of UEs on this
328 DIMM located in channel 0.
329
330
331Channel 1 DIMM Label control file:
332
333 'ch1_dimm_label'
334
335 This control file allows this DIMM to have a label assigned
336 to it. With this label in the module, when errors occur
337 the output can provide the DIMM label in the system log.
338 This becomes vital for panic events to isolate the
339 cause of the UE event.
340
341 DIMM Labels must be assigned after booting, with information
342 that correctly identifies the physical slot with its
343 silk screen label. This information is currently very
344 motherboard specific and determination of this information
345 must occur in userland at this time.
346
Alan Coxda9bb1d2006-01-18 17:44:13 -0800347
Borislav Petkov043b4312015-06-19 11:47:17 +0200348
349SYSTEM LOGGING
350--------------
351
352If logging for UEs and CEs is enabled, then system logs will contain
353information indicating that errors have been detected:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800354
Doug Thompson49c0dab72006-07-10 04:45:19 -0700355EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800356channel 1 "DIMM_B1": amd76x_edac
357
Doug Thompson49c0dab72006-07-10 04:45:19 -0700358EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800359channel 1 "DIMM_B1": amd76x_edac
360
361
362The structure of the message is:
363 the memory controller (MC0)
364 Error type (CE)
365 memory page (0x283)
366 offset in the page (0xce0)
367 the byte granularity (grain 8)
368 or resolution of the error
369 the error syndrome (0xb741)
370 memory row (row 0)
371 memory channel (channel 1)
372 DIMM label, if set prior (DIMM B1
373 and then an optional, driver-specific message that may
374 have additional information.
375
Borislav Petkov043b4312015-06-19 11:47:17 +0200376Both UEs and CEs with no info will lack all but memory controller, error
377type, a notice of "no info" and then an optional, driver-specific error
378message.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800379
380
Alan Coxda9bb1d2006-01-18 17:44:13 -0800381PCI Bus Parity Detection
Borislav Petkov043b4312015-06-19 11:47:17 +0200382------------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800383
Borislav Petkov043b4312015-06-19 11:47:17 +0200384On Header Type 00 devices, the primary status is looked at for any
385parity error regardless of whether parity is enabled on the device or
386not. (The spec indicates parity is generated in some cases). On Header
387Type 01 bridges, the secondary status register is also looked at to see
388if parity occurred on the bus on the other side of the bridge.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800389
390
391SYSFS CONFIGURATION
Borislav Petkov043b4312015-06-19 11:47:17 +0200392-------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800393
394Under /sys/devices/system/edac/pci are control and attribute files as follows:
395
396
397Enable/Disable PCI Parity checking control file:
398
399 'check_pci_parity'
400
401
402 This control file enables or disables the PCI Bus Parity scanning
403 operation. Writing a 1 to this file enables the scanning. Writing
404 a 0 to this file disables the scanning.
405
406 Enable:
407 echo "1" >/sys/devices/system/edac/pci/check_pci_parity
408
409 Disable:
410 echo "0" >/sys/devices/system/edac/pci/check_pci_parity
411
412
Arthur Jones327dafb2008-07-25 01:49:10 -0700413Parity Count:
414
415 'pci_parity_count'
416
417 This attribute file will display the number of parity errors that
418 have been detected.
419
420
Borislav Petkov043b4312015-06-19 11:47:17 +0200421
Arthur Jones327dafb2008-07-25 01:49:10 -0700422MODULE PARAMETERS
Borislav Petkov043b4312015-06-19 11:47:17 +0200423-----------------
Arthur Jones327dafb2008-07-25 01:49:10 -0700424
425Panic on UE control file:
426
427 'edac_mc_panic_on_ue'
428
429 An uncorrectable error will cause a machine panic. This is usually
430 desirable. It is a bad idea to continue when an uncorrectable error
431 occurs - it is indeterminate what was uncorrected and the operating
432 system context might be so mangled that continuing will lead to further
433 corruption. If the kernel has MCE configured, then EDAC will never
434 notice the UE.
435
436 LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1]
437
438 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
439
440
441Log UE control file:
442
443 'edac_mc_log_ue'
444
445 Generate kernel messages describing uncorrectable errors. These errors
446 are reported through the system message log system. UE statistics
447 will be accumulated even when UE logging is disabled.
448
449 LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1]
450
451 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
452
453
454Log CE control file:
455
456 'edac_mc_log_ce'
457
458 Generate kernel messages describing correctable errors. These
459 errors are reported through the system message log system.
460 CE statistics will be accumulated even when CE logging is disabled.
461
462 LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1]
463
464 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
465
466
467Polling period control file:
468
469 'edac_mc_poll_msec'
470
471 The time period, in milliseconds, for polling for error information.
472 Too small a value wastes resources. Too large a value might delay
473 necessary handling of errors and might loose valuable information for
474 locating the error. 1000 milliseconds (once each second) is the current
475 default. Systems which require all the bandwidth they can get, may
476 increase this.
477
478 LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1]
479
480 RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
481
Alan Coxda9bb1d2006-01-18 17:44:13 -0800482
483Panic on PCI PARITY Error:
484
485 'panic_on_pci_parity'
486
487
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300488 This control file enables or disables panicking when a parity
Alan Coxda9bb1d2006-01-18 17:44:13 -0800489 error has been detected.
490
491
Arthur Jones327dafb2008-07-25 01:49:10 -0700492 module/kernel parameter: edac_panic_on_pci_pe=[0|1]
Alan Coxda9bb1d2006-01-18 17:44:13 -0800493
494 Enable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700495 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800496
497 Disable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700498 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800499
500
501
Borislav Petkov043b4312015-06-19 11:47:17 +0200502EDAC device type
503----------------
Doug Thompson87f24c32007-07-19 01:50:34 -0700504
505In the header file, edac_core.h, there is a series of edac_device structures
506and APIs for the EDAC_DEVICE.
507
508User space access to an edac_device is through the sysfs interface.
509
510At the location /sys/devices/system/edac (sysfs) new edac_device devices will
511appear.
512
513There is a three level tree beneath the above 'edac' directory. For example,
514the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
515installs itself as:
516
517 /sys/devices/systm/edac/test-instance
518
519in this directory are various controls, a symlink and one or more 'instance'
Carlos Garciac98be0c2014-04-04 22:31:00 -0400520directories.
Doug Thompson87f24c32007-07-19 01:50:34 -0700521
522The standard default controls are:
523
524 log_ce boolean to log CE events
525 log_ue boolean to log UE events
526 panic_on_ue boolean to 'panic' the system if an UE is encountered
527 (default off, can be set true via startup script)
528 poll_msec time period between POLL cycles for events
529
530The test_device_edac device adds at least one of its own custom control:
531
532 test_bits which in the current test driver does nothing but
533 show how it is installed. A ported driver can
534 add one or more such controls and/or attributes
535 for specific uses.
536 One out-of-tree driver uses controls here to allow
537 for ERROR INJECTION operations to hardware
538 injection registers
539
540The symlink points to the 'struct dev' that is registered for this edac_device.
541
542INSTANCES
Borislav Petkov043b4312015-06-19 11:47:17 +0200543---------
Doug Thompson87f24c32007-07-19 01:50:34 -0700544
545One or more instance directories are present. For the 'test_device_edac' case:
546
547 test-instance0
548
549
550In this directory there are two default counter attributes, which are totals of
551counter in deeper subdirectories.
552
553 ce_count total of CE events of subdirectories
554 ue_count total of UE events of subdirectories
555
556BLOCKS
Borislav Petkov043b4312015-06-19 11:47:17 +0200557------
Doug Thompson87f24c32007-07-19 01:50:34 -0700558
559At the lowest directory level is the 'block' directory. There can be 0, 1
560or more blocks specified in each instance.
561
562 test-block0
563
564
565In this directory the default attributes are:
566
567 ce_count which is counter of CE events for this 'block'
568 of hardware being monitored
569 ue_count which is counter of UE events for this 'block'
570 of hardware being monitored
571
572
573The 'test_device_edac' device adds 4 attributes and 1 control:
574
575 test-block-bits-0 for every POLL cycle this counter
576 is incremented
577 test-block-bits-1 every 10 cycles, this counter is bumped once,
578 and test-block-bits-0 is set to 0
579 test-block-bits-2 every 100 cycles, this counter is bumped once,
580 and test-block-bits-1 is set to 0
581 test-block-bits-3 every 1000 cycles, this counter is bumped once,
582 and test-block-bits-2 is set to 0
583
584
585 reset-counters writing ANY thing to this control will
586 reset all the above counters.
587
588
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300589Use of the 'test_device_edac' driver should enable any others to create their own
Doug Thompson87f24c32007-07-19 01:50:34 -0700590unique drivers for their hardware systems.
591
592The 'test_device_edac' sample driver is located at the
593bluesmoke.sourceforge.net project site for EDAC.
594
Borislav Petkov043b4312015-06-19 11:47:17 +0200595
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300596NEHALEM USAGE OF EDAC APIs
Borislav Petkov043b4312015-06-19 11:47:17 +0200597--------------------------
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300598
599This chapter documents some EXPERIMENTAL mappings for EDAC API to handle
600Nehalem EDAC driver. They will likely be changed on future versions
601of the driver.
602
603Due to the way Nehalem exports Memory Controller data, some adjustments
604were done at i7core_edac driver. This chapter will cover those differences
605
Rami Rosen3aae9ed2015-06-19 09:18:34 +03006061) On Nehalem, there is one Memory Controller per Quick Patch Interconnect
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300607 (QPI). At the driver, the term "socket" means one QPI. This is
608 associated with a physical CPU socket.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300609
610 Each MC have 3 physical read channels, 3 physical write channels and
Masanari Iidac94bed8e2012-04-10 00:22:13 +0900611 3 logic channels. The driver currently sees it as just 3 channels.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300612 Each channel can have up to 3 DIMMs.
613
614 The minimum known unity is DIMMs. There are no information about csrows.
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300615 As EDAC API maps the minimum unity is csrows, the driver sequentially
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300616 maps channel/dimm into different csrows.
617
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300618 For example, supposing the following layout:
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300619 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
620 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
621 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
622 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
623 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
624 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
625 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
626 The driver will map it as:
627 csrow0: channel 0, dimm0
628 csrow1: channel 0, dimm1
629 csrow2: channel 1, dimm0
630 csrow3: channel 2, dimm0
631
632exports one
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300633 DIMM per csrow.
634
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300635 Each QPI is exported as a different memory controller.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300636
Rami Rosen3aae9ed2015-06-19 09:18:34 +03006372) Nehalem MC has the ability to generate errors. The driver implements this
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300638 functionality via some error injection nodes:
639
640 For injecting a memory error, there are some sysfs nodes, under
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300641 /sys/devices/system/edac/mc/mc?/:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300642
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300643 inject_addrmatch/*:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300644 Controls the error injection mask register. It is possible to specify
645 several characteristics of the address to match an error code:
646 dimm = the affected dimm. Numbers are relative to a channel;
647 rank = the memory rank;
648 channel = the channel that will generate an error;
649 bank = the affected bank;
650 page = the page address;
651 column (or col) = the address column.
652 each of the above values can be set to "any" to match any valid value.
653
654 At driver init, all values are set to any.
655
656 For example, to generate an error at rank 1 of dimm 2, for any channel,
657 any bank, any page, any column:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300658 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
659 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300660
661 To return to the default behaviour of matching any, you can do:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300662 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
663 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300664
665 inject_eccmask:
666 specifies what bits will have troubles,
667
668 inject_section:
669 specifies what ECC cache section will get the error:
670 3 for both
671 2 for the highest
672 1 for the lowest
673
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300674 inject_type:
675 specifies the type of error, being a combination of the following bits:
676 bit 0 - repeat
677 bit 1 - ecc
678 bit 2 - parity
679
680 inject_enable starts the error generation when something different
681 than 0 is written.
682
683 All inject vars can be read. root permission is needed for write.
684
685 Datasheet states that the error will only be generated after a write on an
686 address that matches inject_addrmatch. It seems, however, that reading will
687 also produce an error.
688
689 For example, the following code will generate an error for any write access
690 at socket 0, on any DIMM/address on channel 2:
691
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300692 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300693 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
694 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
695 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300696 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
697 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
698
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300699 For socket 1, it is needed to replace "mc0" by "mc1" at the above
700 commands.
701
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300702 The generated error message will look like:
703
704 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
705
7063) Nehalem specific Corrected Error memory counters
707
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300708 Nehalem have some registers to count memory errors. The driver uses those
709 registers to report Corrected Errors on devices with Registered Dimms.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300710
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300711 However, those counters don't work with Unregistered Dimms. As the chipset
712 offers some counters that also work with UDIMMS (but with a worse level of
713 granularity than the default ones), the driver exposes those registers for
714 UDIMM memories.
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300715
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300716 They can be read by looking at the contents of all_channel_counts/
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300717
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300718 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
719 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
720 0
721 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
722 0
723 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
724 0
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300725
726 What happens here is that errors on different csrows, but at the same
727 dimm number will increment the same counter.
728 So, in this memory mapping:
729 csrow0: channel 0, dimm0
730 csrow1: channel 0, dimm1
731 csrow2: channel 1, dimm0
732 csrow3: channel 2, dimm0
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300733 The hardware will increment udimm0 for an error at the first dimm at either
734 csrow0, csrow2 or csrow3;
735 The hardware will increment udimm1 for an error at the second dimm at either
736 csrow0, csrow2 or csrow3;
737 The hardware will increment udimm2 for an error at the third dimm at either
738 csrow0, csrow2 or csrow3;
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300739
7404) Standard error counters
741
742 The standard error counters are generated when an mcelog error is received
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300743 by the driver. Since, with udimm, this is counted by software, it is
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300744 possible that some errors could be lost. With rdimm's, they display the
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300745 contents of the registers
Borislav Petkov043b4312015-06-19 11:47:17 +0200746
Aravind Gopalakrishnan6b7464b2015-09-28 06:44:31 -0500747AMD64_EDAC REFERENCE DOCUMENTS USED
748-----------------------------------
749amd64_edac module is based on the following documents
750(available from http://support.amd.com/en-us/search/tech-docs):
751
7521. Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
753 Opteron Processors
754 AMD publication #: 26094
755 Revision: 3.26
756 Link: http://support.amd.com/TechDocs/26094.PDF
757
7582. Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
759 Processors
760 AMD publication #: 32559
761 Revision: 3.00
762 Issue Date: May 2006
763 Link: http://support.amd.com/TechDocs/32559.pdf
764
7653. Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
766 Processors
767 AMD publication #: 31116
768 Revision: 3.00
769 Issue Date: September 07, 2007
770 Link: http://support.amd.com/TechDocs/31116.pdf
771
7724. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
773 Models 30h-3Fh Processors
774 AMD publication #: 49125
775 Revision: 3.06
776 Issue Date: 2/12/2015 (latest release)
777 Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
778
7795. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
780 Models 60h-6Fh Processors
781 AMD publication #: 50742
782 Revision: 3.01
783 Issue Date: 7/23/2015 (latest release)
784 Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
785
7866. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
787 Models 00h-0Fh Processors
788 AMD publication #: 48751
789 Revision: 3.03
790 Issue Date: 2/23/2015 (latest release)
791 Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
792
Borislav Petkov043b4312015-06-19 11:47:17 +0200793CREDITS:
794========
795
796Written by Doug Thompson <dougthompson@xmission.com>
7977 Dec 2005
79817 Jul 2007 Updated
799
800(c) Mauro Carvalho Chehab
80105 Aug 2009 Nehalem interface
802
803EDAC authors/maintainers:
804
805 Doug Thompson, Dave Jiang, Dave Peterson et al,
806 Mauro Carvalho Chehab
807 Borislav Petkov
808 original author: Thayne Harbaugh