Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 1 | Multi-touch (MT) Protocol |
| 2 | ------------------------- |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 3 | Copyright (C) 2009-2010 Henrik Rydberg <rydberg@euromail.se> |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 4 | |
| 5 | |
| 6 | Introduction |
| 7 | ------------ |
| 8 | |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 9 | In order to utilize the full power of the new multi-touch and multi-user |
| 10 | devices, a way to report detailed data from multiple contacts, i.e., |
| 11 | objects in direct contact with the device surface, is needed. This |
| 12 | document describes the multi-touch (MT) protocol which allows kernel |
| 13 | drivers to report details for an arbitrary number of contacts. |
| 14 | |
| 15 | The protocol is divided into two types, depending on the capabilities of the |
| 16 | hardware. For devices handling anonymous contacts (type A), the protocol |
| 17 | describes how to send the raw data for all contacts to the receiver. For |
| 18 | devices capable of tracking identifiable contacts (type B), the protocol |
| 19 | describes how to send updates for individual contacts via event slots. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 20 | |
| 21 | |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 22 | Protocol Usage |
| 23 | -------------- |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 24 | |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 25 | Contact details are sent sequentially as separate packets of ABS_MT |
| 26 | events. Only the ABS_MT events are recognized as part of a contact |
| 27 | packet. Since these events are ignored by current single-touch (ST) |
| 28 | applications, the MT protocol can be implemented on top of the ST protocol |
| 29 | in an existing driver. |
| 30 | |
| 31 | Drivers for type A devices separate contact packets by calling |
| 32 | input_mt_sync() at the end of each packet. This generates a SYN_MT_REPORT |
| 33 | event, which instructs the receiver to accept the data for the current |
| 34 | contact and prepare to receive another. |
| 35 | |
| 36 | Drivers for type B devices separate contact packets by calling |
| 37 | input_mt_slot(), with a slot as argument, at the beginning of each packet. |
| 38 | This generates an ABS_MT_SLOT event, which instructs the receiver to |
| 39 | prepare for updates of the given slot. |
| 40 | |
| 41 | All drivers mark the end of a multi-touch transfer by calling the usual |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 42 | input_sync() function. This instructs the receiver to act upon events |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 43 | accumulated since last EV_SYN/SYN_REPORT and prepare to receive a new set |
| 44 | of events/packets. |
| 45 | |
| 46 | The main difference between the stateless type A protocol and the stateful |
| 47 | type B slot protocol lies in the usage of identifiable contacts to reduce |
| 48 | the amount of data sent to userspace. The slot protocol requires the use of |
| 49 | the ABS_MT_TRACKING_ID, either provided by the hardware or computed from |
| 50 | the raw data [5]. |
| 51 | |
| 52 | For type A devices, the kernel driver should generate an arbitrary |
| 53 | enumeration of the full set of anonymous contacts currently on the |
| 54 | surface. The order in which the packets appear in the event stream is not |
| 55 | important. Event filtering and finger tracking is left to user space [3]. |
| 56 | |
| 57 | For type B devices, the kernel driver should associate a slot with each |
| 58 | identified contact, and use that slot to propagate changes for the contact. |
| 59 | Creation, replacement and destruction of contacts is achieved by modifying |
| 60 | the ABS_MT_TRACKING_ID of the associated slot. A non-negative tracking id |
| 61 | is interpreted as a contact, and the value -1 denotes an unused slot. A |
| 62 | tracking id not previously present is considered new, and a tracking id no |
| 63 | longer present is considered removed. Since only changes are propagated, |
| 64 | the full state of each initiated contact has to reside in the receiving |
| 65 | end. Upon receiving an MT event, one simply updates the appropriate |
| 66 | attribute of the current slot. |
| 67 | |
Daniel Kurtz | a93bd15 | 2011-08-23 23:02:36 -0700 | [diff] [blame^] | 68 | Some devices identify and/or track more contacts than they can report to the |
| 69 | driver. A driver for such a device should associate one type B slot with each |
| 70 | contact that is reported by the hardware. Whenever the identity of the |
| 71 | contact associated with a slot changes, the driver should invalidate that |
| 72 | slot by changing its ABS_MT_TRACKING_ID. If the hardware signals that it is |
| 73 | tracking more contacts than it is currently reporting, the driver should use |
| 74 | a BTN_TOOL_*TAP event to inform userspace of the total number of contacts |
| 75 | being tracked by the hardware at that moment. The driver should do this by |
| 76 | explicitly sending the corresponding BTN_TOOL_*TAP event and setting |
| 77 | use_count to false when calling input_mt_report_pointer_emulation(). |
| 78 | The driver should only advertise as many slots as the hardware can report. |
| 79 | Userspace can detect that a driver can report more total contacts than slots |
| 80 | by noting that the largest supported BTN_TOOL_*TAP event is larger than the |
| 81 | total number of type B slots reported in the absinfo for the ABS_MT_SLOT axis. |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 82 | |
| 83 | Protocol Example A |
| 84 | ------------------ |
| 85 | |
| 86 | Here is what a minimal event sequence for a two-contact touch would look |
| 87 | like for a type A device: |
| 88 | |
| 89 | ABS_MT_POSITION_X x[0] |
| 90 | ABS_MT_POSITION_Y y[0] |
| 91 | SYN_MT_REPORT |
| 92 | ABS_MT_POSITION_X x[1] |
| 93 | ABS_MT_POSITION_Y y[1] |
| 94 | SYN_MT_REPORT |
| 95 | SYN_REPORT |
| 96 | |
| 97 | The sequence after moving one of the contacts looks exactly the same; the |
| 98 | raw data for all present contacts are sent between every synchronization |
| 99 | with SYN_REPORT. |
| 100 | |
| 101 | Here is the sequence after lifting the first contact: |
| 102 | |
| 103 | ABS_MT_POSITION_X x[1] |
| 104 | ABS_MT_POSITION_Y y[1] |
| 105 | SYN_MT_REPORT |
| 106 | SYN_REPORT |
| 107 | |
| 108 | And here is the sequence after lifting the second contact: |
| 109 | |
| 110 | SYN_MT_REPORT |
| 111 | SYN_REPORT |
| 112 | |
| 113 | If the driver reports one of BTN_TOUCH or ABS_PRESSURE in addition to the |
| 114 | ABS_MT events, the last SYN_MT_REPORT event may be omitted. Otherwise, the |
| 115 | last SYN_REPORT will be dropped by the input core, resulting in no |
| 116 | zero-contact event reaching userland. |
| 117 | |
| 118 | |
| 119 | Protocol Example B |
| 120 | ------------------ |
| 121 | |
| 122 | Here is what a minimal event sequence for a two-contact touch would look |
| 123 | like for a type B device: |
| 124 | |
| 125 | ABS_MT_SLOT 0 |
| 126 | ABS_MT_TRACKING_ID 45 |
| 127 | ABS_MT_POSITION_X x[0] |
| 128 | ABS_MT_POSITION_Y y[0] |
| 129 | ABS_MT_SLOT 1 |
| 130 | ABS_MT_TRACKING_ID 46 |
| 131 | ABS_MT_POSITION_X x[1] |
| 132 | ABS_MT_POSITION_Y y[1] |
| 133 | SYN_REPORT |
| 134 | |
| 135 | Here is the sequence after moving contact 45 in the x direction: |
| 136 | |
| 137 | ABS_MT_SLOT 0 |
| 138 | ABS_MT_POSITION_X x[0] |
| 139 | SYN_REPORT |
| 140 | |
| 141 | Here is the sequence after lifting the contact in slot 0: |
| 142 | |
| 143 | ABS_MT_TRACKING_ID -1 |
| 144 | SYN_REPORT |
| 145 | |
| 146 | The slot being modified is already 0, so the ABS_MT_SLOT is omitted. The |
| 147 | message removes the association of slot 0 with contact 45, thereby |
| 148 | destroying contact 45 and freeing slot 0 to be reused for another contact. |
| 149 | |
| 150 | Finally, here is the sequence after lifting the second contact: |
| 151 | |
| 152 | ABS_MT_SLOT 1 |
| 153 | ABS_MT_TRACKING_ID -1 |
| 154 | SYN_REPORT |
| 155 | |
| 156 | |
| 157 | Event Usage |
| 158 | ----------- |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 159 | |
| 160 | A set of ABS_MT events with the desired properties is defined. The events |
| 161 | are divided into categories, to allow for partial implementation. The |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 162 | minimum set consists of ABS_MT_POSITION_X and ABS_MT_POSITION_Y, which |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 163 | allows for multiple contacts to be tracked. If the device supports it, the |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 164 | ABS_MT_TOUCH_MAJOR and ABS_MT_WIDTH_MAJOR may be used to provide the size |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 165 | of the contact area and approaching contact, respectively. |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 166 | |
| 167 | The TOUCH and WIDTH parameters have a geometrical interpretation; imagine |
| 168 | looking through a window at someone gently holding a finger against the |
| 169 | glass. You will see two regions, one inner region consisting of the part |
| 170 | of the finger actually touching the glass, and one outer region formed by |
| 171 | the perimeter of the finger. The diameter of the inner region is the |
| 172 | ABS_MT_TOUCH_MAJOR, the diameter of the outer region is |
| 173 | ABS_MT_WIDTH_MAJOR. Now imagine the person pressing the finger harder |
| 174 | against the glass. The inner region will increase, and in general, the |
| 175 | ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR, which is always smaller than |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 176 | unity, is related to the contact pressure. For pressure-based devices, |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 177 | ABS_MT_PRESSURE may be used to provide the pressure on the contact area |
Henrik Rydberg | e42a98b | 2010-12-06 10:05:43 +0100 | [diff] [blame] | 178 | instead. Devices capable of contact hovering can use ABS_MT_DISTANCE to |
| 179 | indicate the distance between the contact and the surface. |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 180 | |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 181 | In addition to the MAJOR parameters, the oval shape of the contact can be |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 182 | described by adding the MINOR parameters, such that MAJOR and MINOR are the |
| 183 | major and minor axis of an ellipse. Finally, the orientation of the oval |
| 184 | shape can be describe with the ORIENTATION parameter. |
| 185 | |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 186 | For type A devices, further specification of the touch shape is possible |
| 187 | via ABS_MT_BLOB_ID. |
| 188 | |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 189 | The ABS_MT_TOOL_TYPE may be used to specify whether the touching tool is a |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 190 | finger or a pen or something else. Finally, the ABS_MT_TRACKING_ID event |
| 191 | may be used to track identified contacts over time [5]. |
| 192 | |
| 193 | In the type B protocol, ABS_MT_TOOL_TYPE and ABS_MT_TRACKING_ID are |
| 194 | implicitly handled by input core; drivers should instead call |
| 195 | input_mt_report_slot_state(). |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 196 | |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 197 | |
| 198 | Event Semantics |
| 199 | --------------- |
| 200 | |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 201 | ABS_MT_TOUCH_MAJOR |
| 202 | |
| 203 | The length of the major axis of the contact. The length should be given in |
| 204 | surface units. If the surface has an X times Y resolution, the largest |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 205 | possible value of ABS_MT_TOUCH_MAJOR is sqrt(X^2 + Y^2), the diagonal [4]. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 206 | |
| 207 | ABS_MT_TOUCH_MINOR |
| 208 | |
| 209 | The length, in surface units, of the minor axis of the contact. If the |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 210 | contact is circular, this event can be omitted [4]. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 211 | |
| 212 | ABS_MT_WIDTH_MAJOR |
| 213 | |
| 214 | The length, in surface units, of the major axis of the approaching |
| 215 | tool. This should be understood as the size of the tool itself. The |
| 216 | orientation of the contact and the approaching tool are assumed to be the |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 217 | same [4]. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 218 | |
| 219 | ABS_MT_WIDTH_MINOR |
| 220 | |
| 221 | The length, in surface units, of the minor axis of the approaching |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 222 | tool. Omit if circular [4]. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 223 | |
| 224 | The above four values can be used to derive additional information about |
| 225 | the contact. The ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR approximates |
| 226 | the notion of pressure. The fingers of the hand and the palm all have |
| 227 | different characteristic widths [1]. |
| 228 | |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 229 | ABS_MT_PRESSURE |
| 230 | |
| 231 | The pressure, in arbitrary units, on the contact area. May be used instead |
| 232 | of TOUCH and WIDTH for pressure-based devices or any device with a spatial |
| 233 | signal intensity distribution. |
| 234 | |
Henrik Rydberg | e42a98b | 2010-12-06 10:05:43 +0100 | [diff] [blame] | 235 | ABS_MT_DISTANCE |
| 236 | |
| 237 | The distance, in surface units, between the contact and the surface. Zero |
| 238 | distance means the contact is touching the surface. A positive number means |
| 239 | the contact is hovering above the surface. |
| 240 | |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 241 | ABS_MT_ORIENTATION |
| 242 | |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 243 | The orientation of the ellipse. The value should describe a signed quarter |
| 244 | of a revolution clockwise around the touch center. The signed value range |
| 245 | is arbitrary, but zero should be returned for a finger aligned along the Y |
| 246 | axis of the surface, a negative value when finger is turned to the left, and |
| 247 | a positive value when finger turned to the right. When completely aligned with |
| 248 | the X axis, the range max should be returned. Orientation can be omitted |
| 249 | if the touching object is circular, or if the information is not available |
| 250 | in the kernel driver. Partial orientation support is possible if the device |
| 251 | can distinguish between the two axis, but not (uniquely) any values in |
| 252 | between. In such cases, the range of ABS_MT_ORIENTATION should be [0, 1] |
| 253 | [4]. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 254 | |
| 255 | ABS_MT_POSITION_X |
| 256 | |
| 257 | The surface X coordinate of the center of the touching ellipse. |
| 258 | |
| 259 | ABS_MT_POSITION_Y |
| 260 | |
| 261 | The surface Y coordinate of the center of the touching ellipse. |
| 262 | |
| 263 | ABS_MT_TOOL_TYPE |
| 264 | |
| 265 | The type of approaching tool. A lot of kernel drivers cannot distinguish |
| 266 | between different tool types, such as a finger or a pen. In such cases, the |
| 267 | event should be omitted. The protocol currently supports MT_TOOL_FINGER and |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 268 | MT_TOOL_PEN [2]. For type B devices, this event is handled by input core; |
| 269 | drivers should instead use input_mt_report_slot_state(). |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 270 | |
| 271 | ABS_MT_BLOB_ID |
| 272 | |
| 273 | The BLOB_ID groups several packets together into one arbitrarily shaped |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 274 | contact. The sequence of points forms a polygon which defines the shape of |
| 275 | the contact. This is a low-level anonymous grouping for type A devices, and |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 276 | should not be confused with the high-level trackingID [5]. Most type A |
| 277 | devices do not have blob capability, so drivers can safely omit this event. |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 278 | |
| 279 | ABS_MT_TRACKING_ID |
| 280 | |
| 281 | The TRACKING_ID identifies an initiated contact throughout its life cycle |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 282 | [5]. The value range of the TRACKING_ID should be large enough to ensure |
| 283 | unique identification of a contact maintained over an extended period of |
| 284 | time. For type B devices, this event is handled by input core; drivers |
| 285 | should instead use input_mt_report_slot_state(). |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 286 | |
| 287 | |
| 288 | Event Computation |
| 289 | ----------------- |
| 290 | |
| 291 | The flora of different hardware unavoidably leads to some devices fitting |
| 292 | better to the MT protocol than others. To simplify and unify the mapping, |
| 293 | this section gives recipes for how to compute certain events. |
| 294 | |
| 295 | For devices reporting contacts as rectangular shapes, signed orientation |
| 296 | cannot be obtained. Assuming X and Y are the lengths of the sides of the |
| 297 | touching rectangle, here is a simple formula that retains the most |
| 298 | information possible: |
| 299 | |
| 300 | ABS_MT_TOUCH_MAJOR := max(X, Y) |
| 301 | ABS_MT_TOUCH_MINOR := min(X, Y) |
| 302 | ABS_MT_ORIENTATION := bool(X > Y) |
| 303 | |
| 304 | The range of ABS_MT_ORIENTATION should be set to [0, 1], to indicate that |
| 305 | the device can distinguish between a finger along the Y axis (0) and a |
| 306 | finger along the X axis (1). |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 307 | |
| 308 | |
| 309 | Finger Tracking |
| 310 | --------------- |
| 311 | |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 312 | The process of finger tracking, i.e., to assign a unique trackingID to each |
Henrik Rydberg | 72c8a94 | 2010-07-15 23:22:07 -0700 | [diff] [blame] | 313 | initiated contact on the surface, is a Euclidian Bipartite Matching |
| 314 | problem. At each event synchronization, the set of actual contacts is |
| 315 | matched to the set of contacts from the previous synchronization. A full |
| 316 | implementation can be found in [3]. |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 317 | |
| 318 | |
Henrik Rydberg | f6bdc23 | 2010-01-28 22:28:28 -0800 | [diff] [blame] | 319 | Gestures |
| 320 | -------- |
| 321 | |
| 322 | In the specific application of creating gesture events, the TOUCH and WIDTH |
| 323 | parameters can be used to, e.g., approximate finger pressure or distinguish |
| 324 | between index finger and thumb. With the addition of the MINOR parameters, |
| 325 | one can also distinguish between a sweeping finger and a pointing finger, |
| 326 | and with ORIENTATION, one can detect twisting of fingers. |
| 327 | |
| 328 | |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 329 | Notes |
| 330 | ----- |
| 331 | |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 332 | In order to stay compatible with existing applications, the data reported |
| 333 | in a finger packet must not be recognized as single-touch events. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 334 | |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 335 | For type A devices, all finger data bypasses input filtering, since |
| 336 | subsequent events of the same type refer to different fingers. |
| 337 | |
| 338 | For example usage of the type A protocol, see the bcm5974 driver. For |
| 339 | example usage of the type B protocol, see the hid-egalax driver. |
Henrik Rydberg | eacaad0 | 2009-04-28 07:49:21 -0700 | [diff] [blame] | 340 | |
| 341 | [1] With the extension ABS_MT_APPROACH_X and ABS_MT_APPROACH_Y, the |
| 342 | difference between the contact position and the approaching tool position |
| 343 | could be used to derive tilt. |
| 344 | [2] The list can of course be extended. |
Henrik Rydberg | 22f075a | 2010-12-20 15:09:27 +0100 | [diff] [blame] | 345 | [3] The mtdev project: http://bitmath.org/code/mtdev/. |
Henrik Rydberg | f9fcfc3 | 2009-05-23 09:51:21 -0700 | [diff] [blame] | 346 | [4] See the section on event computation. |
| 347 | [5] See the section on finger tracking. |