Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 1 | This document describes the multiplexing protocol used by ssh(1)'s |
| 2 | ControlMaster connection-sharing. |
| 3 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 4 | Multiplexing starts with a ssh(1) configured to act as a multiplexing |
| 5 | master. This will cause ssh(1) to listen on a Unix domain socket for |
| 6 | requests from clients. Clients communicate over this socket using a |
| 7 | simple packetised protocol, where each message is proceeded with |
| 8 | a length and message type in SSH uint32 wire format: |
| 9 | |
| 10 | uint32 packet length |
| 11 | uint32 packet type |
| 12 | ... packet body |
| 13 | |
| 14 | Most messages from the client to the server contain a "request id" |
| 15 | field. This field is returned in replies as "client request id" to |
| 16 | facilitate matching of responses to requests. |
| 17 | |
| 18 | Many muliplexing (mux) client requests yield immediate responses from |
| 19 | the mux process; requesting a forwarding, performing an alive check or |
| 20 | requesting the master terminate itself fall in to this category. |
| 21 | |
| 22 | The most common use of multiplexing however is to maintain multiple |
| 23 | concurrent sessions. These are supported via two separate modes: |
| 24 | |
| 25 | "Passenger" clients start by requesting a new session with a |
| 26 | MUX_C_NEW_SESSION message and passing stdio file descriptors over the |
| 27 | Unix domain control socket. The passenger client then waits until it is |
| 28 | signaled or the mux server closes the session. This mode is so named as |
| 29 | the client waits around while the mux server does all the driving. |
| 30 | |
| 31 | Stdio forwarding (requested using MUX_C_NEW_STDIO_FWD) is another |
| 32 | example of passenger mode; the client passes the stdio file descriptors |
| 33 | and passively waits for something to happen. |
| 34 | |
| 35 | "Proxy" clients, requested using MUX_C_PROXY, work quite differently. In |
| 36 | this mode, the mux client/server connection socket will stop speaking |
| 37 | the multiplexing protocol and start proxying SSH connection protocol |
| 38 | messages between the client and server. The client therefore must |
| 39 | speak a significant subset of the SSH protocol, but in return is able |
| 40 | to access basically the full suite of connection protocol features. |
| 41 | Moreover, as no file descriptor passing is required, the connection |
| 42 | supporting a proxy client may iteself be forwarded or relayed to another |
| 43 | host if necessary. |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 44 | |
| 45 | 1. Connection setup |
| 46 | |
| 47 | When a multiplexing connection is made to a ssh(1) operating as a |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 48 | ControlMaster from a client ssh(1), the first action of each is send |
| 49 | a hello messages to its peer: |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 50 | |
| 51 | uint32 MUX_MSG_HELLO |
| 52 | uint32 protocol version |
| 53 | string extension name [optional] |
| 54 | string extension value [optional] |
| 55 | ... |
| 56 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 57 | The current version of the mux protocol is 4. A client should refuse |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 58 | to connect to a master that speaks an unsupported protocol version. |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 59 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 60 | Following the version identifier are zero or more extensions represented |
| 61 | as a name/value pair. No extensions are currently defined. |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 62 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 63 | 2. Opening a passenger mode session |
| 64 | |
| 65 | To open a new multiplexed session in passenger mode, a client sends the |
| 66 | following request: |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 67 | |
Damien Miller | 42747df | 2011-01-14 12:01:50 +1100 | [diff] [blame] | 68 | uint32 MUX_C_NEW_SESSION |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 69 | uint32 request id |
| 70 | string reserved |
| 71 | bool want tty flag |
| 72 | bool want X11 forwarding flag |
| 73 | bool want agent flag |
| 74 | bool subsystem flag |
| 75 | uint32 escape char |
| 76 | string terminal type |
| 77 | string command |
| 78 | string environment string 0 [optional] |
| 79 | ... |
| 80 | |
| 81 | To disable the use of an escape character, "escape char" may be set |
| 82 | to 0xffffffff. "terminal type" is generally set to the value of |
| 83 | $TERM. zero or more environment strings may follow the command. |
| 84 | |
| 85 | The client then sends its standard input, output and error file |
| 86 | descriptors (in that order) using Unix domain socket control messages. |
| 87 | |
| 88 | The contents of "reserved" are currently ignored. |
| 89 | |
| 90 | If successful, the server will reply with MUX_S_SESSION_OPENED |
| 91 | |
| 92 | uint32 MUX_S_SESSION_OPENED |
| 93 | uint32 client request id |
| 94 | uint32 session id |
| 95 | |
| 96 | Otherwise it will reply with an error: MUX_S_PERMISSION_DENIED or |
| 97 | MUX_S_FAILURE. |
| 98 | |
| 99 | Once the server has received the fds, it will respond with MUX_S_OK |
| 100 | indicating that the session is up. The client now waits for the |
| 101 | session to end. When it does, the server will send an exit status |
| 102 | message: |
| 103 | |
| 104 | uint32 MUX_S_EXIT_MESSAGE |
| 105 | uint32 session id |
| 106 | uint32 exit value |
| 107 | |
| 108 | The client should exit with this value to mimic the behaviour of a |
| 109 | non-multiplexed ssh(1) connection. Two additional cases that the |
| 110 | client must cope with are it receiving a signal itself and the |
| 111 | server disconnecting without sending an exit message. |
| 112 | |
Damien Miller | 555f3b8 | 2011-05-15 08:48:05 +1000 | [diff] [blame] | 113 | A master may also send a MUX_S_TTY_ALLOC_FAIL before MUX_S_EXIT_MESSAGE |
| 114 | if remote TTY allocation was unsuccessful. The client may use this to |
| 115 | return its local tty to "cooked" mode. |
| 116 | |
| 117 | uint32 MUX_S_TTY_ALLOC_FAIL |
| 118 | uint32 session id |
| 119 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 120 | 3. Requesting passenger-mode stdio forwarding |
| 121 | |
| 122 | A client may request the master to establish a stdio forwarding: |
| 123 | |
| 124 | uint32 MUX_C_NEW_STDIO_FWD |
| 125 | uint32 request id |
| 126 | string reserved |
| 127 | string connect host |
| 128 | string connect port |
| 129 | |
| 130 | The client then sends its standard input and output file descriptors |
| 131 | (in that order) using Unix domain socket control messages. |
| 132 | |
| 133 | The contents of "reserved" are currently ignored. |
| 134 | |
| 135 | A server may reply with a MUX_S_SESSION_OPENED, a MUX_S_PERMISSION_DENIED |
| 136 | or a MUX_S_FAILURE. |
| 137 | |
| 138 | 4. Health checks |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 139 | |
| 140 | The client may request a health check/PID report from a server: |
| 141 | |
| 142 | uint32 MUX_C_ALIVE_CHECK |
| 143 | uint32 request id |
| 144 | |
| 145 | The server replies with: |
| 146 | |
| 147 | uint32 MUX_S_ALIVE |
| 148 | uint32 client request id |
| 149 | uint32 server pid |
| 150 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 151 | 5. Remotely terminating a master |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 152 | |
| 153 | A client may request that a master terminate immediately: |
| 154 | |
| 155 | uint32 MUX_C_TERMINATE |
| 156 | uint32 request id |
| 157 | |
| 158 | The server will reply with one of MUX_S_OK or MUX_S_PERMISSION_DENIED. |
| 159 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 160 | 6. Requesting establishment of port forwards |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 161 | |
| 162 | A client may request the master to establish a port forward: |
| 163 | |
Damien Miller | 42747df | 2011-01-14 12:01:50 +1100 | [diff] [blame] | 164 | uint32 MUX_C_OPEN_FWD |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 165 | uint32 request id |
| 166 | uint32 forwarding type |
| 167 | string listen host |
Damien Miller | 7f12157 | 2012-06-20 21:51:29 +1000 | [diff] [blame] | 168 | uint32 listen port |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 169 | string connect host |
Damien Miller | 7f12157 | 2012-06-20 21:51:29 +1000 | [diff] [blame] | 170 | uint32 connect port |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 171 | |
| 172 | forwarding type may be MUX_FWD_LOCAL, MUX_FWD_REMOTE, MUX_FWD_DYNAMIC. |
| 173 | |
djm@openbsd.org | 356b61f | 2015-07-17 03:04:27 +0000 | [diff] [blame] | 174 | If listen port is (unsigned int) -2, then the listen host is treated as |
| 175 | a unix socket path name. |
| 176 | |
| 177 | If connect port is (unsigned int) -2, then the connect host is treated |
| 178 | as a unix socket path name. |
| 179 | |
Damien Miller | 388f6fc | 2010-05-21 14:57:35 +1000 | [diff] [blame] | 180 | A server may reply with a MUX_S_OK, a MUX_S_REMOTE_PORT, a |
| 181 | MUX_S_PERMISSION_DENIED or a MUX_S_FAILURE. |
| 182 | |
| 183 | For dynamically allocated listen port the server replies with |
| 184 | |
| 185 | uint32 MUX_S_REMOTE_PORT |
| 186 | uint32 client request id |
| 187 | uint32 allocated remote listen port |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 188 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 189 | 7. Requesting closure of port forwards |
Damien Miller | 42747df | 2011-01-14 12:01:50 +1100 | [diff] [blame] | 190 | |
| 191 | Note: currently unimplemented (server will always reply with MUX_S_FAILURE). |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 192 | |
Damien Miller | b407dd8 | 2011-02-04 11:46:39 +1100 | [diff] [blame] | 193 | A client may request the master to close a port forward: |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 194 | |
Damien Miller | 42747df | 2011-01-14 12:01:50 +1100 | [diff] [blame] | 195 | uint32 MUX_C_CLOSE_FWD |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 196 | uint32 request id |
Damien Miller | 4cb855b | 2011-09-22 21:37:38 +1000 | [diff] [blame] | 197 | uint32 forwarding type |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 198 | string listen host |
Damien Miller | 7f12157 | 2012-06-20 21:51:29 +1000 | [diff] [blame] | 199 | uint32 listen port |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 200 | string connect host |
Damien Miller | 7f12157 | 2012-06-20 21:51:29 +1000 | [diff] [blame] | 201 | uint32 connect port |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 202 | |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 203 | A server may reply with a MUX_S_OK, a MUX_S_PERMISSION_DENIED or a |
| 204 | MUX_S_FAILURE. |
| 205 | |
Damien Miller | 6c3eec7 | 2011-05-05 14:16:22 +1000 | [diff] [blame] | 206 | 8. Requesting shutdown of mux listener |
| 207 | |
| 208 | A client may request the master to stop accepting new multiplexing requests |
| 209 | and remove its listener socket. |
| 210 | |
| 211 | uint32 MUX_C_STOP_LISTENING |
| 212 | uint32 request id |
| 213 | |
| 214 | A server may reply with a MUX_S_OK, a MUX_S_PERMISSION_DENIED or a |
| 215 | MUX_S_FAILURE. |
| 216 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 217 | 9. Requesting proxy mode |
| 218 | |
| 219 | A client may request that the the control connection be placed in proxy |
| 220 | mode: |
| 221 | |
| 222 | uint32 MUX_C_PROXY |
| 223 | uint32 request id |
| 224 | |
| 225 | When a mux master receives this message, it will reply with a |
| 226 | confirmation: |
| 227 | |
| 228 | uint32 MUX_S_PROXY |
| 229 | uint32 request id |
| 230 | |
| 231 | And go into proxy mode. All subsequent data over the connection will |
| 232 | be formatted as unencrypted, unpadded, SSH transport messages: |
| 233 | |
| 234 | uint32 packet length |
| 235 | byte 0 (padding length) |
| 236 | byte packet type |
| 237 | byte[packet length - 2] ... |
| 238 | |
| 239 | The mux master will accept most connection messages and global requests, |
| 240 | and will translate channel identifiers to ensure that the proxy client has |
| 241 | globally unique channel numbers (i.e. a proxy client need not worry about |
| 242 | collisions with other clients). |
| 243 | |
| 244 | 10. Status messages |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 245 | |
| 246 | The MUX_S_OK message is empty: |
| 247 | |
| 248 | uint32 MUX_S_OK |
| 249 | uint32 client request id |
| 250 | |
| 251 | The MUX_S_PERMISSION_DENIED and MUX_S_FAILURE include a reason: |
| 252 | |
| 253 | uint32 MUX_S_PERMISSION_DENIED |
| 254 | uint32 client request id |
| 255 | string reason |
| 256 | |
| 257 | uint32 MUX_S_FAILURE |
| 258 | uint32 client request id |
| 259 | string reason |
| 260 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 261 | 11. Protocol numbers |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 262 | |
| 263 | #define MUX_MSG_HELLO 0x00000001 |
| 264 | #define MUX_C_NEW_SESSION 0x10000002 |
| 265 | #define MUX_C_ALIVE_CHECK 0x10000004 |
| 266 | #define MUX_C_TERMINATE 0x10000005 |
Damien Miller | 42747df | 2011-01-14 12:01:50 +1100 | [diff] [blame] | 267 | #define MUX_C_OPEN_FWD 0x10000006 |
| 268 | #define MUX_C_CLOSE_FWD 0x10000007 |
| 269 | #define MUX_C_NEW_STDIO_FWD 0x10000008 |
Damien Miller | 6c3eec7 | 2011-05-05 14:16:22 +1000 | [diff] [blame] | 270 | #define MUX_C_STOP_LISTENING 0x10000009 |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 271 | #define MUX_S_OK 0x80000001 |
| 272 | #define MUX_S_PERMISSION_DENIED 0x80000002 |
| 273 | #define MUX_S_FAILURE 0x80000003 |
| 274 | #define MUX_S_EXIT_MESSAGE 0x80000004 |
| 275 | #define MUX_S_ALIVE 0x80000005 |
| 276 | #define MUX_S_SESSION_OPENED 0x80000006 |
Damien Miller | 388f6fc | 2010-05-21 14:57:35 +1000 | [diff] [blame] | 277 | #define MUX_S_REMOTE_PORT 0x80000007 |
Damien Miller | 555f3b8 | 2011-05-15 08:48:05 +1000 | [diff] [blame] | 278 | #define MUX_S_TTY_ALLOC_FAIL 0x80000008 |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 279 | |
| 280 | #define MUX_FWD_LOCAL 1 |
| 281 | #define MUX_FWD_REMOTE 2 |
| 282 | #define MUX_FWD_DYNAMIC 3 |
| 283 | |
| 284 | XXX TODO |
| 285 | XXX extended status (e.g. report open channels / forwards) |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 286 | XXX lock (maybe) |
| 287 | XXX watch in/out traffic (pre/post crypto) |
| 288 | XXX inject packet (what about replies) |
| 289 | XXX server->client error/warning notifications |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 290 | XXX send signals via mux |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 291 | XXX ^Z support in passengers |
| 292 | XXX extensions for multi-agent |
| 293 | XXX extensions for multi-X11 |
| 294 | XXX session inspection via master |
| 295 | XXX signals via mux request |
| 296 | XXX list active connections via mux |
Damien Miller | b401f92 | 2010-02-10 10:17:49 +1100 | [diff] [blame] | 297 | |
djm@openbsd.org | 92b61a3 | 2018-09-26 07:30:05 +0000 | [diff] [blame] | 298 | $OpenBSD: PROTOCOL.mux,v 1.11 2018/09/26 07:30:05 djm Exp $ |