Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 1 | ATA over Ethernet is a network protocol that provides simple access to |
| 2 | block storage on the LAN. |
| 3 | |
| 4 | http://support.coraid.com/documents/AoEr11.txt |
| 5 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 6 | The EtherDrive (R) HOWTO for 2.6 and 3.x kernels is found at ... |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 7 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 8 | http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 9 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 10 | It has many tips and hints! Please see, especially, recommended |
| 11 | tunings for virtual memory: |
| 12 | |
| 13 | http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO-5.html#ss5.19 |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 14 | |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 15 | The aoetools are userland programs that are designed to work with this |
| 16 | driver. The aoetools are on sourceforge. |
| 17 | |
| 18 | http://aoetools.sourceforge.net/ |
| 19 | |
| 20 | The scripts in this Documentation/aoe directory are intended to |
| 21 | document the use of the driver and are not necessary if you install |
| 22 | the aoetools. |
| 23 | |
| 24 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 25 | CREATING DEVICE NODES |
| 26 | |
| 27 | Users of udev should find the block device nodes created |
| 28 | automatically, but to create all the necessary device nodes, use the |
| 29 | udev configuration rules provided in udev.txt (in this directory). |
| 30 | |
| 31 | There is a udev-install.sh script that shows how to install these |
| 32 | rules on your system. |
| 33 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 34 | There is also an autoload script that shows how to edit |
Lucas De Marchi | 970e248 | 2012-03-30 13:37:16 -0700 | [diff] [blame] | 35 | /etc/modprobe.d/aoe.conf to ensure that the aoe module is loaded when |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 36 | necessary. Preloading the aoe module is preferable to autoloading, |
| 37 | however, because AoE discovery takes a few seconds. It can be |
| 38 | confusing when an AoE device is not present the first time the a |
| 39 | command is run but appears a second later. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 40 | |
| 41 | USING DEVICE NODES |
| 42 | |
| 43 | "cat /dev/etherd/err" blocks, waiting for error diagnostic output, |
| 44 | like any retransmitted packets. |
| 45 | |
| 46 | "echo eth2 eth4 > /dev/etherd/interfaces" tells the aoe driver to |
| 47 | limit ATA over Ethernet traffic to eth2 and eth4. AoE traffic from |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 48 | untrusted networks should be ignored as a matter of security. See |
| 49 | also the aoe_iflist driver option described below. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 50 | |
| 51 | "echo > /dev/etherd/discover" tells the driver to find out what AoE |
| 52 | devices are available. |
| 53 | |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 54 | In the future these character devices may disappear and be replaced |
| 55 | by sysfs counterparts. Using the commands in aoetools insulates |
| 56 | users from these implementation details. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 57 | |
| 58 | The block devices are named like this: |
| 59 | |
| 60 | e{shelf}.{slot} |
| 61 | e{shelf}.{slot}p{part} |
| 62 | |
| 63 | ... so that "e0.2" is the third blade from the left (slot 2) in the |
| 64 | first shelf (shelf address zero). That's the whole disk. The first |
| 65 | partition on that disk would be "e0.2p1". |
| 66 | |
| 67 | USING SYSFS |
| 68 | |
| 69 | Each aoe block device in /sys/block has the extra attributes of |
| 70 | state, mac, and netif. The state attribute is "up" when the device |
| 71 | is ready for I/O and "down" if detected but unusable. The |
| 72 | "down,closewait" state shows that the device is still open and |
| 73 | cannot come up again until it has been closed. |
| 74 | |
| 75 | The mac attribute is the ethernet address of the remote AoE device. |
| 76 | The netif attribute is the network interface on the localhost |
| 77 | through which we are communicating with the remote AoE device. |
| 78 | |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 79 | There is a script in this directory that formats this information in |
| 80 | a convenient way. Users with aoetools should use the aoe-stat |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 81 | command. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 82 | |
| 83 | root@makki root# sh Documentation/aoe/status.sh |
| 84 | e10.0 eth3 up |
| 85 | e10.1 eth3 up |
| 86 | e10.2 eth3 up |
| 87 | e10.3 eth3 up |
| 88 | e10.4 eth3 up |
| 89 | e10.5 eth3 up |
| 90 | e10.6 eth3 up |
| 91 | e10.7 eth3 up |
| 92 | e10.8 eth3 up |
| 93 | e10.9 eth3 up |
| 94 | e4.0 eth1 up |
| 95 | e4.1 eth1 up |
| 96 | e4.2 eth1 up |
| 97 | e4.3 eth1 up |
| 98 | e4.4 eth1 up |
| 99 | e4.5 eth1 up |
| 100 | e4.6 eth1 up |
| 101 | e4.7 eth1 up |
| 102 | e4.8 eth1 up |
| 103 | e4.9 eth1 up |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 104 | |
| 105 | Use /sys/module/aoe/parameters/aoe_iflist (or better, the driver |
| 106 | option discussed below) instead of /dev/etherd/interfaces to limit |
| 107 | AoE traffic to the network interfaces in the given |
| 108 | whitespace-separated list. Unlike the old character device, the |
| 109 | sysfs entry can be read from as well as written to. |
| 110 | |
| 111 | It's helpful to trigger discovery after setting the list of allowed |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 112 | interfaces. The aoetools package provides an aoe-discover script |
| 113 | for this purpose. You can also directly use the |
| 114 | /dev/etherd/discover special file described above. |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 115 | |
| 116 | DRIVER OPTIONS |
| 117 | |
| 118 | There is a boot option for the built-in aoe driver and a |
| 119 | corresponding module parameter, aoe_iflist. Without this option, |
| 120 | all network interfaces may be used for ATA over Ethernet. Here is a |
| 121 | usage example for the module parameter. |
| 122 | |
| 123 | modprobe aoe_iflist="eth1 eth3" |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 124 | |
| 125 | The aoe_deadsecs module parameter determines the maximum number of |
| 126 | seconds that the driver will wait for an AoE device to provide a |
| 127 | response to an AoE command. After aoe_deadsecs seconds have |
| 128 | elapsed, the AoE device will be marked as "down". |
| 129 | |
| 130 | The aoe_maxout module parameter has a default of 128. This is the |
| 131 | maximum number of unresponded packets that will be sent to an AoE |
| 132 | target at one time. |
| 133 | |
| 134 | The aoe_dyndevs module parameter defaults to 1, meaning that the |
| 135 | driver will assign a block device minor number to a discovered AoE |
| 136 | target based on the order of its discovery. With dynamic minor |
| 137 | device numbers in use, a greater range of AoE shelf and slot |
| 138 | addresses can be supported. Users with udev will never have to |
| 139 | think about minor numbers. Using aoe_dyndevs=0 allows device nodes |
| 140 | to be pre-created using a static minor-number scheme with the |
| 141 | aoe-mkshelf script in the aoetools. |