Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 1 | Short users guide for SLUB |
| 2 | -------------------------- |
| 3 | |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 4 | The basic philosophy of SLUB is very different from SLAB. SLAB |
| 5 | requires rebuilding the kernel to activate debug options for all |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 6 | slab caches. SLUB always includes full debugging but it is off by default. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 7 | SLUB can enable debugging only for selected slabs in order to avoid |
| 8 | an impact on overall system performance which may make a bug more |
| 9 | difficult to find. |
| 10 | |
| 11 | In order to switch debugging on one can add a option "slub_debug" |
| 12 | to the kernel command line. That will enable full debugging for |
| 13 | all slabs. |
| 14 | |
| 15 | Typically one would then use the "slabinfo" command to get statistical |
| 16 | data and perform operation on the slabs. By default slabinfo only lists |
| 17 | slabs that have data in them. See "slabinfo -h" for more options when |
| 18 | running the command. slabinfo can be compiled with |
| 19 | |
majianpeng | 9fe4961 | 2012-05-03 16:34:39 +0800 | [diff] [blame] | 20 | gcc -o slabinfo tools/vm/slabinfo.c |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 21 | |
| 22 | Some of the modes of operation of slabinfo require that slub debugging |
| 23 | be enabled on the command line. F.e. no tracking information will be |
| 24 | available without debugging on and validation can only partially |
| 25 | be performed if debugging was not switched on. |
| 26 | |
| 27 | Some more sophisticated uses of slub_debug: |
| 28 | ------------------------------------------- |
| 29 | |
| 30 | Parameters may be given to slub_debug. If none is specified then full |
| 31 | debugging is enabled. Format: |
| 32 | |
| 33 | slub_debug=<Debug-Options> Enable options for all slabs |
| 34 | slub_debug=<Debug-Options>,<slab name> |
| 35 | Enable options only for select slabs |
| 36 | |
| 37 | Possible debug options are |
| 38 | F Sanity checks on (enables SLAB_DEBUG_FREE. Sorry |
| 39 | SLAB legacy issues) |
| 40 | Z Red zoning |
| 41 | P Poisoning (object and padding) |
| 42 | U User tracking (free and alloc) |
| 43 | T Trace (please only use on single slabs) |
Dmitry Monakhov | 4c13dd3 | 2010-02-26 09:36:12 +0300 | [diff] [blame] | 44 | A Toggle failslab filter mark for the cache |
David Rientjes | fa5ec8a | 2009-07-07 00:14:14 -0700 | [diff] [blame] | 45 | O Switch debugging off for caches that would have |
| 46 | caused higher minimum slab orders |
Christoph Lameter | f0630ff | 2007-07-15 23:38:14 -0700 | [diff] [blame] | 47 | - Switch all debugging off (useful if the kernel is |
| 48 | configured with CONFIG_SLUB_DEBUG_ON) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 49 | |
| 50 | F.e. in order to boot just with sanity checks and red zoning one would specify: |
| 51 | |
| 52 | slub_debug=FZ |
| 53 | |
| 54 | Trying to find an issue in the dentry cache? Try |
| 55 | |
Itaru Kitayama | 989a724 | 2008-03-05 15:07:30 -0800 | [diff] [blame] | 56 | slub_debug=,dentry |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 57 | |
| 58 | to only enable debugging on the dentry cache. |
| 59 | |
| 60 | Red zoning and tracking may realign the slab. We can just apply sanity checks |
| 61 | to the dentry cache with |
| 62 | |
Itaru Kitayama | 989a724 | 2008-03-05 15:07:30 -0800 | [diff] [blame] | 63 | slub_debug=F,dentry |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 64 | |
David Rientjes | fa5ec8a | 2009-07-07 00:14:14 -0700 | [diff] [blame] | 65 | Debugging options may require the minimum possible slab order to increase as |
| 66 | a result of storing the metadata (for example, caches with PAGE_SIZE object |
| 67 | sizes). This has a higher liklihood of resulting in slab allocation errors |
| 68 | in low memory situations or if there's high fragmentation of memory. To |
| 69 | switch off debugging for such caches by default, use |
| 70 | |
| 71 | slub_debug=O |
| 72 | |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 73 | In case you forgot to enable debugging on the kernel command line: It is |
| 74 | possible to enable debugging manually when the kernel is up. Look at the |
| 75 | contents of: |
| 76 | |
Greg Kroah-Hartman | 081248d | 2007-11-01 09:29:06 -0600 | [diff] [blame] | 77 | /sys/kernel/slab/<slab name>/ |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 78 | |
| 79 | Look at the writable files. Writing 1 to them will enable the |
| 80 | corresponding debug option. All options can be set on a slab that does |
| 81 | not contain objects. If the slab already contains objects then sanity checks |
| 82 | and tracing may only be enabled. The other options may cause the realignment |
| 83 | of objects. |
| 84 | |
| 85 | Careful with tracing: It may spew out lots of information and never stop if |
| 86 | used on the wrong slab. |
| 87 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 88 | Slab merging |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 89 | ------------ |
| 90 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 91 | If no debug options are specified then SLUB may merge similar slabs together |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 92 | in order to reduce overhead and increase cache hotness of objects. |
| 93 | slabinfo -a displays which slabs were merged together. |
| 94 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 95 | Slab validation |
| 96 | --------------- |
| 97 | |
| 98 | SLUB can validate all object if the kernel was booted with slub_debug. In |
| 99 | order to do so you must have the slabinfo tool. Then you can do |
| 100 | |
| 101 | slabinfo -v |
| 102 | |
| 103 | which will test all objects. Output will be generated to the syslog. |
| 104 | |
| 105 | This also works in a more limited way if boot was without slab debug. |
| 106 | In that case slabinfo -v simply tests all reachable objects. Usually |
| 107 | these are in the cpu slabs and the partial slabs. Full slabs are not |
| 108 | tracked by SLUB in a non debug situation. |
| 109 | |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 110 | Getting more performance |
| 111 | ------------------------ |
| 112 | |
| 113 | To some degree SLUB's performance is limited by the need to take the |
| 114 | list_lock once in a while to deal with partial slabs. That overhead is |
| 115 | governed by the order of the allocation for each slab. The allocations |
| 116 | can be influenced by kernel parameters: |
| 117 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 118 | slub_min_objects=x (default 4) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 119 | slub_min_order=x (default 0) |
Eric Dumazet | 25f4379 | 2011-11-22 06:59:09 +0100 | [diff] [blame] | 120 | slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 121 | |
| 122 | slub_min_objects allows to specify how many objects must at least fit |
| 123 | into one slab in order for the allocation order to be acceptable. |
| 124 | In general slub will be able to perform this number of allocations |
| 125 | on a slab without consulting centralized resources (list_lock) where |
| 126 | contention may occur. |
| 127 | |
| 128 | slub_min_order specifies a minim order of slabs. A similar effect like |
| 129 | slub_min_objects. |
| 130 | |
| 131 | slub_max_order specified the order at which slub_min_objects should no |
| 132 | longer be checked. This is useful to avoid SLUB trying to generate |
| 133 | super large order pages to fit slub_min_objects of a slab cache with |
Stanislaw Gruszka | 888a214 | 2012-01-12 17:17:39 -0800 | [diff] [blame] | 134 | large object sizes into one high order page. Setting command line |
| 135 | parameter debug_guardpage_minorder=N (N > 0), forces setting |
| 136 | slub_max_order to 0, what cause minimum possible order of slabs |
| 137 | allocation. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 138 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 139 | SLUB Debug output |
| 140 | ----------------- |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 141 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 142 | Here is a sample of slub debug output: |
| 143 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 144 | ==================================================================== |
| 145 | BUG kmalloc-8: Redzone overwritten |
| 146 | -------------------------------------------------------------------- |
| 147 | |
| 148 | INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc |
| 149 | INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 |
| 150 | INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 |
| 151 | INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 |
| 152 | |
| 153 | Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ |
| 154 | Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005 |
| 155 | Redzone 0xc90f6d28: 00 cc cc cc . |
| 156 | Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ |
| 157 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 158 | [<c010523d>] dump_trace+0x63/0x1eb |
| 159 | [<c01053df>] show_trace_log_lvl+0x1a/0x2f |
| 160 | [<c010601d>] show_trace+0x12/0x14 |
| 161 | [<c0106035>] dump_stack+0x16/0x18 |
| 162 | [<c017e0fa>] object_err+0x143/0x14b |
| 163 | [<c017e2cc>] check_object+0x66/0x234 |
| 164 | [<c017eb43>] __slab_free+0x239/0x384 |
| 165 | [<c017f446>] kfree+0xa6/0xc6 |
| 166 | [<c02e2335>] get_modalias+0xb9/0xf5 |
| 167 | [<c02e23b7>] dmi_dev_uevent+0x27/0x3c |
| 168 | [<c027866a>] dev_uevent+0x1ad/0x1da |
| 169 | [<c0205024>] kobject_uevent_env+0x20a/0x45b |
| 170 | [<c020527f>] kobject_uevent+0xa/0xf |
| 171 | [<c02779f1>] store_uevent+0x4f/0x58 |
| 172 | [<c027758e>] dev_attr_store+0x29/0x2f |
| 173 | [<c01bec4f>] sysfs_write_file+0x16e/0x19c |
| 174 | [<c0183ba7>] vfs_write+0xd1/0x15a |
| 175 | [<c01841d7>] sys_write+0x3d/0x72 |
| 176 | [<c0104112>] sysenter_past_esp+0x5f/0x99 |
| 177 | [<b7f7b410>] 0xb7f7b410 |
| 178 | ======================= |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 179 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 180 | FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 181 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 182 | If SLUB encounters a corrupted object (full detection requires the kernel |
| 183 | to be booted with slub_debug) then the following output will be dumped |
| 184 | into the syslog: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 185 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 186 | 1. Description of the problem encountered |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 187 | |
| 188 | This will be a message in the system log starting with |
| 189 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 190 | =============================================== |
| 191 | BUG <slab cache affected>: <What went wrong> |
| 192 | ----------------------------------------------- |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 193 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 194 | INFO: <corruption start>-<corruption_end> <more info> |
| 195 | INFO: Slab <address> <slab information> |
| 196 | INFO: Object <address> <object information> |
| 197 | INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by |
| 198 | cpu> pid=<pid of the process> |
| 199 | INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> |
| 200 | pid=<pid of the process> |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 201 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 202 | (Object allocation / free information is only available if SLAB_STORE_USER is |
| 203 | set for the slab. slub_debug sets that option) |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 204 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 205 | 2. The object contents if an object was involved. |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 206 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 207 | Various types of lines can follow the BUG SLUB line: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 208 | |
| 209 | Bytes b4 <address> : <bytes> |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 210 | Shows a few bytes before the object where the problem was detected. |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 211 | Can be useful if the corruption does not stop with the start of the |
| 212 | object. |
| 213 | |
| 214 | Object <address> : <bytes> |
| 215 | The bytes of the object. If the object is inactive then the bytes |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 216 | typically contain poison values. Any non-poison value shows a |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 217 | corruption by a write after free. |
| 218 | |
| 219 | Redzone <address> : <bytes> |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 220 | The Redzone following the object. The Redzone is used to detect |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 221 | writes after the object. All bytes should always have the same |
| 222 | value. If there is any deviation then it is due to a write after |
| 223 | the object boundary. |
| 224 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 225 | (Redzone information is only available if SLAB_RED_ZONE is set. |
| 226 | slub_debug sets that option) |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 227 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 228 | Padding <address> : <bytes> |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 229 | Unused data to fill up the space in order to get the next object |
| 230 | properly aligned. In the debug case we make sure that there are |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 231 | at least 4 bytes of padding. This allows the detection of writes |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 232 | before the object. |
| 233 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 234 | 3. A stackdump |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 235 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 236 | The stackdump describes the location where the error was detected. The cause |
| 237 | of the corruption is may be more likely found by looking at the function that |
| 238 | allocated or freed the object. |
| 239 | |
| 240 | 4. Report on how the problem was dealt with in order to ensure the continued |
| 241 | operation of the system. |
| 242 | |
| 243 | These are messages in the system log beginning with |
| 244 | |
| 245 | FIX <slab cache affected>: <corrective action taken> |
| 246 | |
| 247 | In the above sample SLUB found that the Redzone of an active object has |
| 248 | been overwritten. Here a string of 8 characters was written into a slab that |
| 249 | has the length of 8 characters. However, a 8 character string needs a |
| 250 | terminating 0. That zero has overwritten the first byte of the Redzone field. |
| 251 | After reporting the details of the issue encountered the FIX SLUB message |
Justin P. Mattock | e02f0e8 | 2009-12-04 00:36:35 -0800 | [diff] [blame] | 252 | tells us that SLUB has restored the Redzone to its proper value and then |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 253 | system operations continue. |
| 254 | |
| 255 | Emergency operations: |
| 256 | --------------------- |
| 257 | |
| 258 | Minimal debugging (sanity checks alone) can be enabled by booting with |
| 259 | |
| 260 | slub_debug=F |
| 261 | |
| 262 | This will be generally be enough to enable the resiliency features of slub |
| 263 | which will keep the system running even if a bad kernel component will |
| 264 | keep corrupting objects. This may be important for production systems. |
| 265 | Performance will be impacted by the sanity checks and there will be a |
| 266 | continual stream of error messages to the syslog but no additional memory |
| 267 | will be used (unlike full debugging). |
| 268 | |
| 269 | No guarantees. The kernel component still needs to be fixed. Performance |
| 270 | may be optimized further by locating the slab that experiences corruption |
| 271 | and enabling debugging only for that cache |
| 272 | |
| 273 | I.e. |
| 274 | |
| 275 | slub_debug=F,dentry |
| 276 | |
| 277 | If the corruption occurs by writing after the end of the object then it |
| 278 | may be advisable to enable a Redzone to avoid corrupting the beginning |
| 279 | of other objects. |
| 280 | |
| 281 | slub_debug=FZ,dentry |
| 282 | |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 283 | Extended slabinfo mode and plotting |
| 284 | ----------------------------------- |
| 285 | |
| 286 | The slabinfo tool has a special 'extended' ('-X') mode that includes: |
| 287 | - Slabcache Totals |
| 288 | - Slabs sorted by size (up to -N <num> slabs, default 1) |
| 289 | - Slabs sorted by loss (up to -N <num> slabs, default 1) |
| 290 | |
| 291 | Additionally, in this mode slabinfo does not dynamically scale sizes (G/M/K) |
| 292 | and reports everything in bytes (this functionality is also available to |
| 293 | other slabinfo modes via '-B' option) which makes reporting more precise and |
| 294 | accurate. Moreover, in some sense the `-X' mode also simplifies the analysis |
| 295 | of slabs' behaviour, because its output can be plotted using the |
| 296 | slabinfo-gnuplot.sh script. So it pushes the analysis from looking through |
| 297 | the numbers (tons of numbers) to something easier -- visual analysis. |
| 298 | |
| 299 | To generate plots: |
| 300 | a) collect slabinfo extended records, for example: |
| 301 | |
| 302 | while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done |
| 303 | |
| 304 | b) pass stats file(-s) to slabinfo-gnuplot.sh script: |
| 305 | slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN] |
| 306 | |
| 307 | The slabinfo-gnuplot.sh script will pre-processes the collected records |
| 308 | and generates 3 png files (and 3 pre-processing cache files) per STATS |
| 309 | file: |
| 310 | - Slabcache Totals: FOO_STATS-totals.png |
| 311 | - Slabs sorted by size: FOO_STATS-slabs-by-size.png |
| 312 | - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png |
| 313 | |
| 314 | Another use case, when slabinfo-gnuplot can be useful, is when you need |
| 315 | to compare slabs' behaviour "prior to" and "after" some code modification. |
| 316 | To help you out there, slabinfo-gnuplot.sh script can 'merge' the |
| 317 | `Slabcache Totals` sections from different measurements. To visually |
| 318 | compare N plots: |
| 319 | |
| 320 | a) Collect as many STATS1, STATS2, .. STATSN files as you need |
| 321 | while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done |
| 322 | |
| 323 | b) Pre-process those STATS files |
| 324 | slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN |
| 325 | |
| 326 | c) Execute slabinfo-gnuplot.sh in '-t' mode, passing all of the |
| 327 | generated pre-processed *-totals |
| 328 | slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals |
| 329 | |
| 330 | This will produce a single plot (png file). |
| 331 | |
| 332 | Plots, expectedly, can be large so some fluctuations or small spikes |
| 333 | can go unnoticed. To deal with that, `slabinfo-gnuplot.sh' has two |
| 334 | options to 'zoom-in'/'zoom-out': |
| 335 | a) -s %d,%d overwrites the default image width and heigh |
| 336 | b) -r %d,%d specifies a range of samples to use (for example, |
| 337 | in `slabinfo -X >> FOO_STATS; sleep 1;' case, using |
| 338 | a "-r 40,60" range will plot only samples collected |
| 339 | between 40th and 60th seconds). |
| 340 | |
Christoph Lameter | cde5353 | 2008-07-04 09:59:22 -0700 | [diff] [blame] | 341 | Christoph Lameter, May 30, 2007 |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 342 | Sergey Senozhatsky, October 23, 2015 |