blob: 844c03fe7921d62539b63c7748bfc120fb7b87e1 [file] [log] [blame]
Linus Torvalds1da177e2005-04-16 15:20:36 -07001
2 Debugging on Linux for s/390 & z/Architecture
3 by
4 Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
5 Copyright (C) 2000-2001 IBM Deutschland Entwicklung GmbH, IBM Corporation
6 Best viewed with fixed width fonts
7
8Overview of Document:
9=====================
10This document is intended to give an good overview of how to debug
11Linux for s/390 & z/Architecture it isn't intended as a complete reference & not a
12tutorial on the fundamentals of C & assembly, it dosen't go into
13390 IO in any detail. It is intended to complement the documents in the
14reference section below & any other worthwhile references you get.
15
16It is intended like the Enterprise Systems Architecture/390 Reference Summary
17to be printed out & used as a quick cheat sheet self help style reference when
18problems occur.
19
20Contents
21========
22Register Set
23Address Spaces on Intel Linux
24Address Spaces on Linux for s/390 & z/Architecture
25The Linux for s/390 & z/Architecture Kernel Task Structure
26Register Usage & Stackframes on Linux for s/390 & z/Architecture
27A sample program with comments
28Compiling programs for debugging on Linux for s/390 & z/Architecture
29Figuring out gcc compile errors
30Debugging Tools
31objdump
32strace
33Performance Debugging
34Debugging under VM
35s/390 & z/Architecture IO Overview
36Debugging IO on s/390 & z/Architecture under VM
37GDB on s/390 & z/Architecture
38Stack chaining in gdb by hand
39Examining core dumps
40ldd
41Debugging modules
42The proc file system
43Starting points for debugging scripting languages etc.
44Dumptool & Lcrash
45SysRq
46References
47Special Thanks
48
49Register Set
50============
51The current architectures have the following registers.
52
5316 General propose registers, 32 bit on s/390 64 bit on z/Architecture, r0-r15 or gpr0-gpr15 used for arithmetic & addressing.
54
5516 Control registers, 32 bit on s/390 64 bit on z/Architecture, ( cr0-cr15 kernel usage only ) used for memory management,
56interrupt control,debugging control etc.
57
5816 Access registers ( ar0-ar15 ) 32 bit on s/390 & z/Architecture
59not used by normal programs but potentially could
60be used as temporary storage. Their main purpose is their 1 to 1
61association with general purpose registers and are used in
62the kernel for copying data between kernel & user address spaces.
63Access register 0 ( & access register 1 on z/Architecture ( needs 64 bit
64pointer ) ) is currently used by the pthread library as a pointer to
65the current running threads private area.
66
6716 64 bit floating point registers (fp0-fp15 ) IEEE & HFP floating
68point format compliant on G5 upwards & a Floating point control reg (FPC)
694 64 bit registers (fp0,fp2,fp4 & fp6) HFP only on older machines.
70Note:
71Linux (currently) always uses IEEE & emulates G5 IEEE format on older machines,
72( provided the kernel is configured for this ).
73
74
75The PSW is the most important register on the machine it
76is 64 bit on s/390 & 128 bit on z/Architecture & serves the roles of
77a program counter (pc), condition code register,memory space designator.
78In IBM standard notation I am counting bit 0 as the MSB.
79It has several advantages over a normal program counter
80in that you can change address translation & program counter
81in a single instruction. To change address translation,
82e.g. switching address translation off requires that you
83have a logical=physical mapping for the address you are
84currently running at.
85
86 Bit Value
87s/390 z/Architecture
880 0 Reserved ( must be 0 ) otherwise specification exception occurs.
89
901 1 Program Event Recording 1 PER enabled,
91 PER is used to facilititate debugging e.g. single stepping.
92
932-4 2-4 Reserved ( must be 0 ).
94
955 5 Dynamic address translation 1=DAT on.
96
976 6 Input/Output interrupt Mask
98
997 7 External interrupt Mask used primarily for interprocessor signalling &
100 clock interrupts.
101
1028-11 8-11 PSW Key used for complex memory protection mechanism not used under linux
103
10412 12 1 on s/390 0 on z/Architecture
105
10613 13 Machine Check Mask 1=enable machine check interrupts
107
10814 14 Wait State set this to 1 to stop the processor except for interrupts & give
109 time to other LPARS used in CPU idle in the kernel to increase overall
110 usage of processor resources.
111
11215 15 Problem state ( if set to 1 certain instructions are disabled )
113 all linux user programs run with this bit 1
114 ( useful info for debugging under VM ).
115
11616-17 16-17 Address Space Control
117
118 00 Primary Space Mode when DAT on
119 The linux kernel currently runs in this mode, CR1 is affiliated with
120 this mode & points to the primary segment table origin etc.
121
122 01 Access register mode this mode is used in functions to
123 copy data between kernel & user space.
124
125 10 Secondary space mode not used in linux however CR7 the
126 register affiliated with this mode is & this & normally
127 CR13=CR7 to allow us to copy data between kernel & user space.
128 We do this as follows:
129 We set ar2 to 0 to designate its
130 affiliated gpr ( gpr2 )to point to primary=kernel space.
131 We set ar4 to 1 to designate its
132 affiliated gpr ( gpr4 ) to point to secondary=home=user space
133 & then essentially do a memcopy(gpr2,gpr4,size) to
134 copy data between the address spaces, the reason we use home space for the
135 kernel & don't keep secondary space free is that code will not run in
136 secondary space.
137
138 11 Home Space Mode all user programs run in this mode.
139 it is affiliated with CR13.
140
14118-19 18-19 Condition codes (CC)
142
14320 20 Fixed point overflow mask if 1=FPU exceptions for this event
144 occur ( normally 0 )
145
14621 21 Decimal overflow mask if 1=FPU exceptions for this event occur
147 ( normally 0 )
148
14922 22 Exponent underflow mask if 1=FPU exceptions for this event occur
150 ( normally 0 )
151
15223 23 Significance Mask if 1=FPU exceptions for this event occur
153 ( normally 0 )
154
15524-31 24-30 Reserved Must be 0.
156
157 31 Extended Addressing Mode
158 32 Basic Addressing Mode
159 Used to set addressing mode
160 PSW 31 PSW 32
161 0 0 24 bit
162 0 1 31 bit
163 1 1 64 bit
164
16532 1=31 bit addressing mode 0=24 bit addressing mode (for backward
166 compatibility ), linux always runs with this bit set to 1
167
16833-64 Instruction address.
169 33-63 Reserved must be 0
170 64-127 Address
171 In 24 bits mode bits 64-103=0 bits 104-127 Address
172 In 31 bits mode bits 64-96=0 bits 97-127 Address
173 Note: unlike 31 bit mode on s/390 bit 96 must be zero
174 when loading the address with LPSWE otherwise a
175 specification exception occurs, LPSW is fully backward
176 compatible.
177
178
179Prefix Page(s)
180--------------
181This per cpu memory area is too intimately tied to the processor not to mention.
182It exists between the real addresses 0-4096 on s/390 & 0-8192 z/Architecture & is exchanged
183with a 1 page on s/390 or 2 pages on z/Architecture in absolute storage by the set
184prefix instruction in linux'es startup.
185This page is mapped to a different prefix for each processor in an SMP configuration
186( assuming the os designer is sane of course :-) ).
187Bytes 0-512 ( 200 hex ) on s/390 & 0-512,4096-4544,4604-5119 currently on z/Architecture
188are used by the processor itself for holding such information as exception indications &
189entry points for exceptions.
190Bytes after 0xc00 hex are used by linux for per processor globals on s/390 & z/Architecture
191( there is a gap on z/Architecure too currently between 0xc00 & 1000 which linux uses ).
192The closest thing to this on traditional architectures is the interrupt
193vector table. This is a good thing & does simplify some of the kernel coding
194however it means that we now cannot catch stray NULL pointers in the
195kernel without hard coded checks.
196
197
198
199Address Spaces on Intel Linux
200=============================
201
202The traditional Intel Linux is approximately mapped as follows forgive
203the ascii art.
2040xFFFFFFFF 4GB Himem *****************
205 * *
206 * Kernel Space *
207 * *
208 ***************** ****************
209User Space Himem (typically 0xC0000000 3GB )* User Stack * * *
210 ***************** * *
211 * Shared Libs * * Next Process *
212 ***************** * to *
213 * * <== * Run * <==
214 * User Program * * *
215 * Data BSS * * *
216 * Text * * *
217 * Sections * * *
2180x00000000 ***************** ****************
219
220Now it is easy to see that on Intel it is quite easy to recognise a kernel address
221as being one greater than user space himem ( in this case 0xC0000000).
222& addresses of less than this are the ones in the current running program on this
223processor ( if an smp box ).
224If using the virtual machine ( VM ) as a debugger it is quite difficult to
225know which user process is running as the address space you are looking at
226could be from any process in the run queue.
227
228The limitation of Intels addressing technique is that the linux
229kernel uses a very simple real address to virtual addressing technique
230of Real Address=Virtual Address-User Space Himem.
231This means that on Intel the kernel linux can typically only address
232Himem=0xFFFFFFFF-0xC0000000=1GB & this is all the RAM these machines
233can typically use.
234They can lower User Himem to 2GB or lower & thus be
235able to use 2GB of RAM however this shrinks the maximum size
236of User Space from 3GB to 2GB they have a no win limit of 4GB unless
237they go to 64 Bit.
238
239
240On 390 our limitations & strengths make us slightly different.
241For backward compatibility we are only allowed use 31 bits (2GB)
242of our 32 bit addresses,however, we use entirely separate address
243spaces for the user & kernel.
244
245This means we can support 2GB of non Extended RAM on s/390, & more
246with the Extended memory management swap device &
247currently 4TB of physical memory currently on z/Architecture.
248
249
250Address Spaces on Linux for s/390 & z/Architecture
251==================================================
252
253Our addressing scheme is as follows
254
255
256Himem 0x7fffffff 2GB on s/390 ***************** ****************
257currently 0x3ffffffffff (2^42)-1 * User Stack * * *
258on z/Architecture. ***************** * *
259 * Shared Libs * * *
260 ***************** * *
261 * * * Kernel *
262 * User Program * * *
263 * Data BSS * * *
264 * Text * * *
265 * Sections * * *
2660x00000000 ***************** ****************
267
268This also means that we need to look at the PSW problem state bit
269or the addressing mode to decide whether we are looking at
270user or kernel space.
271
272Virtual Addresses on s/390 & z/Architecture
273===========================================
274
275A virtual address on s/390 is made up of 3 parts
276The SX ( segment index, roughly corresponding to the PGD & PMD in linux terminology )
277being bits 1-11.
278The PX ( page index, corresponding to the page table entry (pte) in linux terminology )
279being bits 12-19.
280The remaining bits BX (the byte index are the offset in the page )
281i.e. bits 20 to 31.
282
283On z/Architecture in linux we currently make up an address from 4 parts.
284The region index bits (RX) 0-32 we currently use bits 22-32
285The segment index (SX) being bits 33-43
286The page index (PX) being bits 44-51
287The byte index (BX) being bits 52-63
288
289Notes:
2901) s/390 has no PMD so the PMD is really the PGD also.
291A lot of this stuff is defined in pgtable.h.
292
2932) Also seeing as s/390's page indexes are only 1k in size
294(bits 12-19 x 4 bytes per pte ) we use 1 ( page 4k )
295to make the best use of memory by updating 4 segment indices
296entries each time we mess with a PMD & use offsets
2970,1024,2048 & 3072 in this page as for our segment indexes.
298On z/Architecture our page indexes are now 2k in size
299( bits 12-19 x 8 bytes per pte ) we do a similar trick
300but only mess with 2 segment indices each time we mess with
301a PMD.
302
3033) As z/Architecture supports upto a massive 5-level page table lookup we
304can only use 3 currently on Linux ( as this is all the generic kernel
305currently supports ) however this may change in future
306this allows us to access ( according to my sums )
3074TB of virtual storage per process i.e.
3084096*512(PTES)*1024(PMDS)*2048(PGD) = 4398046511104 bytes,
309enough for another 2 or 3 of years I think :-).
310to do this we use a region-third-table designation type in
311our address space control registers.
312
313
314The Linux for s/390 & z/Architecture Kernel Task Structure
315==========================================================
316Each process/thread under Linux for S390 has its own kernel task_struct
317defined in linux/include/linux/sched.h
318The S390 on initialisation & resuming of a process on a cpu sets
319the __LC_KERNEL_STACK variable in the spare prefix area for this cpu
320( which we use for per processor globals).
321
322The kernel stack pointer is intimately tied with the task stucture for
323each processor as follows.
324
325 s/390
326 ************************
327 * 1 page kernel stack *
328 * ( 4K ) *
329 ************************
330 * 1 page task_struct *
331 * ( 4K ) *
3328K aligned ************************
333
334 z/Architecture
335 ************************
336 * 2 page kernel stack *
337 * ( 8K ) *
338 ************************
339 * 2 page task_struct *
340 * ( 8K ) *
34116K aligned ************************
342
343What this means is that we don't need to dedicate any register or global variable
344to point to the current running process & can retrieve it with the following
345very simple construct for s/390 & one very similar for z/Architecture.
346
347static inline struct task_struct * get_current(void)
348{
349 struct task_struct *current;
350 __asm__("lhi %0,-8192\n\t"
351 "nr %0,15"
352 : "=r" (current) );
353 return current;
354}
355
356i.e. just anding the current kernel stack pointer with the mask -8192.
357Thankfully because Linux dosen't have support for nested IO interrupts
358& our devices have large buffers can survive interrupts being shut for
359short amounts of time we don't need a separate stack for interrupts.
360
361
362
363
364Register Usage & Stackframes on Linux for s/390 & z/Architecture
365=================================================================
366Overview:
367---------
368This is the code that gcc produces at the top & the bottom of
369each function, it usually is fairly consistent & similar from
370function to function & if you know its layout you can probalby
371make some headway in finding the ultimate cause of a problem
372after a crash without a source level debugger.
373
374Note: To follow stackframes requires a knowledge of C or Pascal &
375limited knowledge of one assembly language.
376
377It should be noted that there are some differences between the
378s/390 & z/Architecture stack layouts as the z/Architecture stack layout didn't have
379to maintain compatibility with older linkage formats.
380
381Glossary:
382---------
383alloca:
384This is a built in compiler function for runtime allocation
385of extra space on the callers stack which is obviously freed
386up on function exit ( e.g. the caller may choose to allocate nothing
387of a buffer of 4k if required for temporary purposes ), it generates
388very efficient code ( a few cycles ) when compared to alternatives
389like malloc.
390
391automatics: These are local variables on the stack,
392i.e they aren't in registers & they aren't static.
393
394back-chain:
395This is a pointer to the stack pointer before entering a
396framed functions ( see frameless function ) prologue got by
397deferencing the address of the current stack pointer,
398 i.e. got by accessing the 32 bit value at the stack pointers
399current location.
400
401base-pointer:
402This is a pointer to the back of the literal pool which
403is an area just behind each procedure used to store constants
404in each function.
405
406call-clobbered: The caller probably needs to save these registers if there
407is something of value in them, on the stack or elsewhere before making a
408call to another procedure so that it can restore it later.
409
410epilogue:
411The code generated by the compiler to return to the caller.
412
413frameless-function
414A frameless function in Linux for s390 & z/Architecture is one which doesn't
415need more than the register save area ( 96 bytes on s/390, 160 on z/Architecture )
416given to it by the caller.
417A frameless function never:
4181) Sets up a back chain.
4192) Calls alloca.
4203) Calls other normal functions
4214) Has automatics.
422
423GOT-pointer:
424This is a pointer to the global-offset-table in ELF
425( Executable Linkable Format, Linux'es most common executable format ),
426all globals & shared library objects are found using this pointer.
427
428lazy-binding
429ELF shared libraries are typically only loaded when routines in the shared
430library are actually first called at runtime. This is lazy binding.
431
432procedure-linkage-table
433This is a table found from the GOT which contains pointers to routines
434in other shared libraries which can't be called to by easier means.
435
436prologue:
437The code generated by the compiler to set up the stack frame.
438
439outgoing-args:
440This is extra area allocated on the stack of the calling function if the
441parameters for the callee's cannot all be put in registers, the same
442area can be reused by each function the caller calls.
443
444routine-descriptor:
445A COFF executable format based concept of a procedure reference
446actually being 8 bytes or more as opposed to a simple pointer to the routine.
447This is typically defined as follows
448Routine Descriptor offset 0=Pointer to Function
449Routine Descriptor offset 4=Pointer to Table of Contents
450The table of contents/TOC is roughly equivalent to a GOT pointer.
451& it means that shared libraries etc. can be shared between several
452environments each with their own TOC.
453
454
455static-chain: This is used in nested functions a concept adopted from pascal
456by gcc not used in ansi C or C++ ( although quite useful ), basically it
457is a pointer used to reference local variables of enclosing functions.
458You might come across this stuff once or twice in your lifetime.
459
460e.g.
461The function below should return 11 though gcc may get upset & toss warnings
462about unused variables.
463int FunctionA(int a)
464{
465 int b;
466 FunctionC(int c)
467 {
468 b=c+1;
469 }
470 FunctionC(10);
471 return(b);
472}
473
474
475s/390 & z/Architecture Register usage
476=====================================
477r0 used by syscalls/assembly call-clobbered
478r1 used by syscalls/assembly call-clobbered
479r2 argument 0 / return value 0 call-clobbered
480r3 argument 1 / return value 1 (if long long) call-clobbered
481r4 argument 2 call-clobbered
482r5 argument 3 call-clobbered
483r6 argument 5 saved
484r7 pointer-to arguments 5 to ... saved
485r8 this & that saved
486r9 this & that saved
487r10 static-chain ( if nested function ) saved
488r11 frame-pointer ( if function used alloca ) saved
489r12 got-pointer saved
490r13 base-pointer saved
491r14 return-address saved
492r15 stack-pointer saved
493
494f0 argument 0 / return value ( float/double ) call-clobbered
495f2 argument 1 call-clobbered
496f4 z/Architecture argument 2 saved
497f6 z/Architecture argument 3 saved
498The remaining floating points
499f1,f3,f5 f7-f15 are call-clobbered.
500
501Notes:
502------
5031) The only requirement is that registers which are used
504by the callee are saved, e.g. the compiler is perfectly
505capible of using r11 for purposes other than a frame a
506frame pointer if a frame pointer is not needed.
5072) In functions with variable arguments e.g. printf the calling procedure
508is identical to one without variable arguments & the same number of
509parameters. However, the prologue of this function is somewhat more
510hairy owing to it having to move these parameters to the stack to
511get va_start, va_arg & va_end to work.
5123) Access registers are currently unused by gcc but are used in
513the kernel. Possibilities exist to use them at the moment for
514temporary storage but it isn't recommended.
5154) Only 4 of the floating point registers are used for
516parameter passing as older machines such as G3 only have only 4
517& it keeps the stack frame compatible with other compilers.
518However with IEEE floating point emulation under linux on the
519older machines you are free to use the other 12.
5205) A long long or double parameter cannot be have the
521first 4 bytes in a register & the second four bytes in the
522outgoing args area. It must be purely in the outgoing args
523area if crossing this boundary.
5246) Floating point parameters are mixed with outgoing args
525on the outgoing args area in the order the are passed in as parameters.
5267) Floating point arguments 2 & 3 are saved in the outgoing args area for
527z/Architecture
528
529
530Stack Frame Layout
531------------------
532s/390 z/Architecture
5330 0 back chain ( a 0 here signifies end of back chain )
5344 8 eos ( end of stack, not used on Linux for S390 used in other linkage formats )
5358 16 glue used in other s/390 linkage formats for saved routine descriptors etc.
53612 24 glue used in other s/390 linkage formats for saved routine descriptors etc.
53716 32 scratch area
53820 40 scratch area
53924 48 saved r6 of caller function
54028 56 saved r7 of caller function
54132 64 saved r8 of caller function
54236 72 saved r9 of caller function
54340 80 saved r10 of caller function
54444 88 saved r11 of caller function
54548 96 saved r12 of caller function
54652 104 saved r13 of caller function
54756 112 saved r14 of caller function
54860 120 saved r15 of caller function
54964 128 saved f4 of caller function
55072 132 saved f6 of caller function
55180 undefined
55296 160 outgoing args passed from caller to callee
55396+x 160+x possible stack alignment ( 8 bytes desirable )
55496+x+y 160+x+y alloca space of caller ( if used )
55596+x+y+z 160+x+y+z automatics of caller ( if used )
5560 back-chain
557
558A sample program with comments.
559===============================
560
561Comments on the function test
562-----------------------------
5631) It didn't need to set up a pointer to the constant pool gpr13 as it isn't used
564( :-( ).
5652) This is a frameless function & no stack is bought.
5663) The compiler was clever enough to recognise that it could return the
567value in r2 as well as use it for the passed in parameter ( :-) ).
5684) The basr ( branch relative & save ) trick works as follows the instruction
569has a special case with r0,r0 with some instruction operands is understood as
570the literal value 0, some risc architectures also do this ). So now
571we are branching to the next address & the address new program counter is
572in r13,so now we subtract the size of the function prologue we have executed
573+ the size of the literal pool to get to the top of the literal pool
5740040037c int test(int b)
575{ # Function prologue below
576 40037c: 90 de f0 34 stm %r13,%r14,52(%r15) # Save registers r13 & r14
577 400380: 0d d0 basr %r13,%r0 # Set up pointer to constant pool using
578 400382: a7 da ff fa ahi %r13,-6 # basr trick
579 return(5+b);
580 # Huge main program
581 400386: a7 2a 00 05 ahi %r2,5 # add 5 to r2
582
583 # Function epilogue below
584 40038a: 98 de f0 34 lm %r13,%r14,52(%r15) # restore registers r13 & 14
585 40038e: 07 fe br %r14 # return
586}
587
588Comments on the function main
589-----------------------------
5901) The compiler did this function optimally ( 8-) )
591
592Literal pool for main.
593400390: ff ff ff ec .long 0xffffffec
594main(int argc,char *argv[])
595{ # Function prologue below
596 400394: 90 bf f0 2c stm %r11,%r15,44(%r15) # Save necessary registers
597 400398: 18 0f lr %r0,%r15 # copy stack pointer to r0
598 40039a: a7 fa ff a0 ahi %r15,-96 # Make area for callee saving
599 40039e: 0d d0 basr %r13,%r0 # Set up r13 to point to
600 4003a0: a7 da ff f0 ahi %r13,-16 # literal pool
601 4003a4: 50 00 f0 00 st %r0,0(%r15) # Save backchain
602
603 return(test(5)); # Main Program Below
604 4003a8: 58 e0 d0 00 l %r14,0(%r13) # load relative address of test from
605 # literal pool
606 4003ac: a7 28 00 05 lhi %r2,5 # Set first parameter to 5
607 4003b0: 4d ee d0 00 bas %r14,0(%r14,%r13) # jump to test setting r14 as return
608 # address using branch & save instruction.
609
610 # Function Epilogue below
611 4003b4: 98 bf f0 8c lm %r11,%r15,140(%r15)# Restore necessary registers.
612 4003b8: 07 fe br %r14 # return to do program exit
613}
614
615
616Compiler updates
617----------------
618
619main(int argc,char *argv[])
620{
621 4004fc: 90 7f f0 1c stm %r7,%r15,28(%r15)
622 400500: a7 d5 00 04 bras %r13,400508 <main+0xc>
623 400504: 00 40 04 f4 .long 0x004004f4
624 # compiler now puts constant pool in code to so it saves an instruction
625 400508: 18 0f lr %r0,%r15
626 40050a: a7 fa ff a0 ahi %r15,-96
627 40050e: 50 00 f0 00 st %r0,0(%r15)
628 return(test(5));
629 400512: 58 10 d0 00 l %r1,0(%r13)
630 400516: a7 28 00 05 lhi %r2,5
631 40051a: 0d e1 basr %r14,%r1
632 # compiler adds 1 extra instruction to epilogue this is done to
633 # avoid processor pipeline stalls owing to data dependencies on g5 &
634 # above as register 14 in the old code was needed directly after being loaded
635 # by the lm %r11,%r15,140(%r15) for the br %14.
636 40051c: 58 40 f0 98 l %r4,152(%r15)
637 400520: 98 7f f0 7c lm %r7,%r15,124(%r15)
638 400524: 07 f4 br %r4
639}
640
641
642Hartmut ( our compiler developer ) also has been threatening to take out the
643stack backchain in optimised code as this also causes pipeline stalls, you
644have been warned.
645
64664 bit z/Architecture code disassembly
647--------------------------------------
648
649If you understand the stuff above you'll understand the stuff
650below too so I'll avoid repeating myself & just say that
651some of the instructions have g's on the end of them to indicate
652they are 64 bit & the stack offsets are a bigger,
653the only other difference you'll find between 32 & 64 bit is that
654we now use f4 & f6 for floating point arguments on 64 bit.
65500000000800005b0 <test>:
656int test(int b)
657{
658 return(5+b);
659 800005b0: a7 2a 00 05 ahi %r2,5
660 800005b4: b9 14 00 22 lgfr %r2,%r2 # downcast to integer
661 800005b8: 07 fe br %r14
662 800005ba: 07 07 bcr 0,%r7
663
664
665}
666
66700000000800005bc <main>:
668main(int argc,char *argv[])
669{
670 800005bc: eb bf f0 58 00 24 stmg %r11,%r15,88(%r15)
671 800005c2: b9 04 00 1f lgr %r1,%r15
672 800005c6: a7 fb ff 60 aghi %r15,-160
673 800005ca: e3 10 f0 00 00 24 stg %r1,0(%r15)
674 return(test(5));
675 800005d0: a7 29 00 05 lghi %r2,5
676 # brasl allows jumps > 64k & is overkill here bras would do fune
677 800005d4: c0 e5 ff ff ff ee brasl %r14,800005b0 <test>
678 800005da: e3 40 f1 10 00 04 lg %r4,272(%r15)
679 800005e0: eb bf f0 f8 00 04 lmg %r11,%r15,248(%r15)
680 800005e6: 07 f4 br %r4
681}
682
683
684
685Compiling programs for debugging on Linux for s/390 & z/Architecture
686====================================================================
687-gdwarf-2 now works it should be considered the default debugging
688format for s/390 & z/Architecture as it is more reliable for debugging
689shared libraries, normal -g debugging works much better now
690Thanks to the IBM java compiler developers bug reports.
691
692This is typically done adding/appending the flags -g or -gdwarf-2 to the
693CFLAGS & LDFLAGS variables Makefile of the program concerned.
694
695If using gdb & you would like accurate displays of registers &
696 stack traces compile without optimisation i.e make sure
697that there is no -O2 or similar on the CFLAGS line of the Makefile &
698the emitted gcc commands, obviously this will produce worse code
699( not advisable for shipment ) but it is an aid to the debugging process.
700
701This aids debugging because the compiler will copy parameters passed in
702in registers onto the stack so backtracing & looking at passed in
703parameters will work, however some larger programs which use inline functions
704will not compile without optimisation.
705
706Debugging with optimisation has since much improved after fixing
707some bugs, please make sure you are using gdb-5.0 or later developed
708after Nov'2000.
709
710Figuring out gcc compile errors
711===============================
712If you are getting a lot of syntax errors compiling a program & the problem
713isn't blatantly obvious from the source.
714It often helps to just preprocess the file, this is done with the -E
715option in gcc.
716What this does is that it runs through the very first phase of compilation
717( compilation in gcc is done in several stages & gcc calls many programs to
718achieve its end result ) with the -E option gcc just calls the gcc preprocessor (cpp).
719The c preprocessor does the following, it joins all the files #included together
720recursively ( #include files can #include other files ) & also the c file you wish to compile.
721It puts a fully qualified path of the #included files in a comment & it
722does macro expansion.
723This is useful for debugging because
7241) You can double check whether the files you expect to be included are the ones
725that are being included ( e.g. double check that you aren't going to the i386 asm directory ).
7262) Check that macro definitions aren't clashing with typedefs,
7273) Check that definitons aren't being used before they are being included.
7284) Helps put the line emitting the error under the microscope if it contains macros.
729
730For convenience the Linux kernel's makefile will do preprocessing automatically for you
731by suffixing the file you want built with .i ( instead of .o )
732
733e.g.
734from the linux directory type
735make arch/s390/kernel/signal.i
736this will build
737
738s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
739-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -E arch/s390/kernel/signal.c
740> arch/s390/kernel/signal.i
741
742Now look at signal.i you should see something like.
743
744
745# 1 "/home1/barrow/linux/include/asm/types.h" 1
746typedef unsigned short umode_t;
747typedef __signed__ char __s8;
748typedef unsigned char __u8;
749typedef __signed__ short __s16;
750typedef unsigned short __u16;
751
752If instead you are getting errors further down e.g.
753unknown instruction:2515 "move.l" or better still unknown instruction:2515
754"Fixme not implemented yet, call Martin" you are probably are attempting to compile some code
755meant for another architecture or code that is simply not implemented, with a fixme statement
756stuck into the inline assembly code so that the author of the file now knows he has work to do.
757To look at the assembly emitted by gcc just before it is about to call gas ( the gnu assembler )
758use the -S option.
759Again for your convenience the Linux kernel's Makefile will hold your hand &
760do all this donkey work for you also by building the file with the .s suffix.
761e.g.
762from the Linux directory type
763make arch/s390/kernel/signal.s
764
765s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
766-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -S arch/s390/kernel/signal.c
767-o arch/s390/kernel/signal.s
768
769
770This will output something like, ( please note the constant pool & the useful comments
771in the prologue to give you a hand at interpreting it ).
772
773.LC54:
774 .string "misaligned (__u16 *) in __xchg\n"
775.LC57:
776 .string "misaligned (__u32 *) in __xchg\n"
777.L$PG1: # Pool sys_sigsuspend
778.LC192:
779 .long -262401
780.LC193:
781 .long -1
782.LC194:
783 .long schedule-.L$PG1
784.LC195:
785 .long do_signal-.L$PG1
786 .align 4
787.globl sys_sigsuspend
788 .type sys_sigsuspend,@function
789sys_sigsuspend:
790# leaf function 0
791# automatics 16
792# outgoing args 0
793# need frame pointer 0
794# call alloca 0
795# has varargs 0
796# incoming args (stack) 0
797# function length 168
798 STM 8,15,32(15)
799 LR 0,15
800 AHI 15,-112
801 BASR 13,0
802.L$CO1: AHI 13,.L$PG1-.L$CO1
803 ST 0,0(15)
804 LR 8,2
805 N 5,.LC192-.L$PG1(13)
806
807Adding -g to the above output makes the output even more useful
808e.g. typing
809make CC:="s390-gcc -g" kernel/sched.s
810
811which compiles.
812s390-gcc -g -D__KERNEL__ -I/home/barrow/linux-2.3/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe -fno-strength-reduce -S kernel/sched.c -o kernel/sched.s
813
814also outputs stabs ( debugger ) info, from this info you can find out the
815offsets & sizes of various elements in structures.
816e.g. the stab for the structure
817struct rlimit {
818 unsigned long rlim_cur;
819 unsigned long rlim_max;
820};
821is
822.stabs "rlimit:T(151,2)=s8rlim_cur:(0,5),0,32;rlim_max:(0,5),32,32;;",128,0,0,0
823from this stab you can see that
824rlimit_cur starts at bit offset 0 & is 32 bits in size
825rlimit_max starts at bit offset 32 & is 32 bits in size.
826
827
828Debugging Tools:
829================
830
831objdump
832=======
833This is a tool with many options the most useful being ( if compiled with -g).
834objdump --source <victim program or object file> > <victims debug listing >
835
836
837The whole kernel can be compiled like this ( Doing this will make a 17MB kernel
838& a 200 MB listing ) however you have to strip it before building the image
839using the strip command to make it a more reasonable size to boot it.
840
841A source/assembly mixed dump of the kernel can be done with the line
842objdump --source vmlinux > vmlinux.lst
843Also if the file isn't compiled -g this will output as much debugging information
844as it can ( e.g. function names ), however, this is very slow as it spends lots
845of time searching for debugging info, the following self explanitory line should be used
846instead if the code isn't compiled -g.
847objdump --disassemble-all --syms vmlinux > vmlinux.lst
848as it is much faster
849
850As hard drive space is valuble most of us use the following approach.
8511) Look at the emitted psw on the console to find the crash address in the kernel.
8522) Look at the file System.map ( in the linux directory ) produced when building
853the kernel to find the closest address less than the current PSW to find the
854offending function.
8553) use grep or similar to search the source tree looking for the source file
856 with this function if you don't know where it is.
8574) rebuild this object file with -g on, as an example suppose the file was
858( /arch/s390/kernel/signal.o )
8595) Assuming the file with the erroneous function is signal.c Move to the base of the
860Linux source tree.
8616) rm /arch/s390/kernel/signal.o
8627) make /arch/s390/kernel/signal.o
8638) watch the gcc command line emitted
8649) type it in again or alernatively cut & paste it on the console adding the -g option.
86510) objdump --source arch/s390/kernel/signal.o > signal.lst
866This will output the source & the assembly intermixed, as the snippet below shows
867This will unfortunately output addresses which aren't the same
868as the kernel ones you should be able to get around the mental arithmetic
869by playing with the --adjust-vma parameter to objdump.
870
871
872
873
Adrian Bunk4448aaf2005-11-08 21:34:42 -0800874static inline void spin_lock(spinlock_t *lp)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700875{
876 a0: 18 34 lr %r3,%r4
877 a2: a7 3a 03 bc ahi %r3,956
878 __asm__ __volatile(" lhi 1,-1\n"
879 a6: a7 18 ff ff lhi %r1,-1
880 aa: 1f 00 slr %r0,%r0
881 ac: ba 01 30 00 cs %r0,%r1,0(%r3)
882 b0: a7 44 ff fd jm aa <sys_sigsuspend+0x2e>
883 saveset = current->blocked;
884 b4: d2 07 f0 68 mvc 104(8,%r15),972(%r4)
885 b8: 43 cc
886 return (set->sig[0] & mask) != 0;
887}
888
8896) If debugging under VM go down to that section in the document for more info.
890
891
892I now have a tool which takes the pain out of --adjust-vma
893& you are able to do something like
894make /arch/s390/kernel/traps.lst
895& it automatically generates the correctly relocated entries for
896the text segment in traps.lst.
897This tool is now standard in linux distro's in scripts/makelst
898
899strace:
900-------
901Q. What is it ?
902A. It is a tool for intercepting calls to the kernel & logging them
903to a file & on the screen.
904
905Q. What use is it ?
906A. You can used it to find out what files a particular program opens.
907
908
909
910Example 1
911---------
912If you wanted to know does ping work but didn't have the source
913strace ping -c 1 127.0.0.1
914& then look at the man pages for each of the syscalls below,
915( In fact this is sometimes easier than looking at some spagetti
916source which conditionally compiles for several architectures )
917Not everything that it throws out needs to make sense immeadiately
918
919Just looking quickly you can see that it is making up a RAW socket
920for the ICMP protocol.
921Doing an alarm(10) for a 10 second timeout
922& doing a gettimeofday call before & after each read to see
923how long the replies took, & writing some text to stdout so the user
924has an idea what is going on.
925
926socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
927getuid() = 0
928setuid(0) = 0
929stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
930stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory)
931stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
932getpid() = 353
933setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
934setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0
935fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0
936mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000
937ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
938write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes
939) = 42
940sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0
941sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0
942gettimeofday({948904719, 138951}, NULL) = 0
943sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET,
944sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64
945sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
946sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
947alarm(10) = 0
948recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0,
949{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
950gettimeofday({948904719, 160224}, NULL) = 0
951recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0,
952{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
953gettimeofday({948904719, 166952}, NULL) = 0
954write(1, "64 bytes from 127.0.0.1: icmp_se"...,
9555764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms
956
957Example 2
958---------
959strace passwd 2>&1 | grep open
960produces the following output
961open("/etc/ld.so.cache", O_RDONLY) = 3
962open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
963open("/lib/libc.so.5", O_RDONLY) = 3
964open("/dev", O_RDONLY) = 3
965open("/var/run/utmp", O_RDONLY) = 3
966open("/etc/passwd", O_RDONLY) = 3
967open("/etc/shadow", O_RDONLY) = 3
968open("/etc/login.defs", O_RDONLY) = 4
969open("/dev/tty", O_RDONLY) = 4
970
971The 2>&1 is done to redirect stderr to stdout & grep is then filtering this input
972through the pipe for each line containing the string open.
973
974
975Example 3
976---------
977Getting sophistocated
978telnetd crashes on & I don't know why
979Steps
980-----
9811) Replace the following line in /etc/inetd.conf
982telnet stream tcp nowait root /usr/sbin/in.telnetd -h
983with
984telnet stream tcp nowait root /blah
985
9862) Create the file /blah with the following contents to start tracing telnetd
987#!/bin/bash
988/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h
9893) chmod 700 /blah to make it executable only to root
9904)
991killall -HUP inetd
992or ps aux | grep inetd
993get inetd's process id
994& kill -HUP inetd to restart it.
995
996Important options
997-----------------
998-o is used to tell strace to output to a file in our case t1 in the root directory
999-f is to follow children i.e.
1000e.g in our case above telnetd will start the login process & subsequently a shell like bash.
1001You will be able to tell which is which from the process ID's listed on the left hand side
1002of the strace output.
1003-p<pid> will tell strace to attach to a running process, yup this can be done provided
1004 it isn't being traced or debugged already & you have enough privileges,
1005the reason 2 processes cannot trace or debug the same program is that strace
1006becomes the parent process of the one being debugged & processes ( unlike people )
1007can have only one parent.
1008
1009
1010However the file /t1 will get big quite quickly
1011to test it telnet 127.0.0.1
1012
1013now look at what files in.telnetd execve'd
1014413 execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0
1015414 execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0
1016
1017Whey it worked!.
1018
1019
1020Other hints:
1021------------
1022If the program is not very interactive ( i.e. not much keyboard input )
1023& is crashing in one architecture but not in another you can do
1024an strace of both programs under as identical a scenario as you can
1025on both architectures outputting to a file then.
1026do a diff of the two traces using the diff program
1027i.e.
1028diff output1 output2
1029& maybe you'll be able to see where the call paths differed, this
1030is possibly near the cause of the crash.
1031
1032More info
1033---------
1034Look at man pages for strace & the various syscalls
1035e.g. man strace, man alarm, man socket.
1036
1037
1038Performance Debugging
1039=====================
1040gcc is capible of compiling in profiling code just add the -p option
1041to the CFLAGS, this obviously affects program size & performance.
1042This can be used by the gprof gnu profiling tool or the
1043gcov the gnu code coverage tool ( code coverage is a means of testing
1044code quality by checking if all the code in an executable in exercised by
1045a tester ).
1046
1047
1048Using top to find out where processes are sleeping in the kernel
1049----------------------------------------------------------------
1050To do this copy the System.map from the root directory where
1051the linux kernel was built to the /boot directory on your
1052linux machine.
1053Start top
1054Now type fU<return>
1055You should see a new field called WCHAN which
1056tells you where each process is sleeping here is a typical output.
1057
1058 6:59pm up 41 min, 1 user, load average: 0.00, 0.00, 0.00
105928 processes: 27 sleeping, 1 running, 0 zombie, 0 stopped
1060CPU states: 0.0% user, 0.1% system, 0.0% nice, 99.8% idle
1061Mem: 254900K av, 45976K used, 208924K free, 0K shrd, 28636K buff
1062Swap: 0K av, 0K used, 0K free 8620K cached
1063
1064 PID USER PRI NI SIZE RSS SHARE WCHAN STAT LIB %CPU %MEM TIME COMMAND
1065 750 root 12 0 848 848 700 do_select S 0 0.1 0.3 0:00 in.telnetd
1066 767 root 16 0 1140 1140 964 R 0 0.1 0.4 0:00 top
1067 1 root 8 0 212 212 180 do_select S 0 0.0 0.0 0:00 init
1068 2 root 9 0 0 0 0 down_inte SW 0 0.0 0.0 0:00 kmcheck
1069
1070The time command
1071----------------
1072Another related command is the time command which gives you an indication
1073of where a process is spending the majority of its time.
1074e.g.
1075time ping -c 5 nc
1076outputs
1077real 0m4.054s
1078user 0m0.010s
1079sys 0m0.010s
1080
1081Debugging under VM
1082==================
1083
1084Notes
1085-----
1086Addresses & values in the VM debugger are always hex never decimal
1087Address ranges are of the format <HexValue1>-<HexValue2> or <HexValue1>.<HexValue2>
1088e.g. The address range 0x2000 to 0x3000 can be described described as
10892000-3000 or 2000.1000
1090
1091The VM Debugger is case insensitive.
1092
1093VM's strengths are usually other debuggers weaknesses you can get at any resource
1094no matter how sensitive e.g. memory management resources,change address translation
1095in the PSW. For kernel hacking you will reap dividends if you get good at it.
1096
1097The VM Debugger displays operators but not operands, probably because some
1098of it was written when memory was expensive & the programmer was probably proud that
1099it fitted into 2k of memory & the programmers & didn't want to shock hardcore VM'ers by
1100changing the interface :-), also the debugger displays useful information on the same line &
1101the author of the code probably felt that it was a good idea not to go over
1102the 80 columns on the screen.
1103
1104As some of you are probably in a panic now this isn't as unintuitive as it may seem
1105as the 390 instructions are easy to decode mentally & you can make a good guess at a lot
1106of them as all the operands are nibble ( half byte aligned ) & if you have an objdump listing
1107also it is quite easy to follow, if you don't have an objdump listing keep a copy of
1108the s/390 Reference Summary & look at between pages 2 & 7 or alternatively the
1109s/390 principles of operation.
1110e.g. even I can guess that
11110001AFF8' LR 180F CC 0
1112is a ( load register ) lr r0,r15
1113
1114Also it is very easy to tell the length of a 390 instruction from the 2 most significant
1115bits in the instruction ( not that this info is really useful except if you are trying to
1116make sense of a hexdump of code ).
1117Here is a table
1118Bits Instruction Length
1119------------------------------------------
112000 2 Bytes
112101 4 Bytes
112210 4 Bytes
112311 6 Bytes
1124
1125
1126
1127
1128The debugger also displays other useful info on the same line such as the
1129addresses being operated on destination addresses of branches & condition codes.
1130e.g.
113100019736' AHI A7DAFF0E CC 1
1132000198BA' BRC A7840004 -> 000198C2' CC 0
1133000198CE' STM 900EF068 >> 0FA95E78 CC 2
1134
1135
1136
1137Useful VM debugger commands
1138---------------------------
1139
1140I suppose I'd better mention this before I start
1141to list the current active traces do
1142Q TR
1143there can be a maximum of 255 of these per set
1144( more about trace sets later ).
1145To stop traces issue a
1146TR END.
1147To delete a particular breakpoint issue
1148TR DEL <breakpoint number>
1149
1150The PA1 key drops to CP mode so you can issue debugger commands,
1151Doing alt c (on my 3270 console at least ) clears the screen.
1152hitting b <enter> comes back to the running operating system
1153from cp mode ( in our case linux ).
1154It is typically useful to add shortcuts to your profile.exec file
1155if you have one ( this is roughly equivalent to autoexec.bat in DOS ).
1156file here are a few from mine.
1157/* this gives me command history on issuing f12 */
1158set pf12 retrieve
1159/* this continues */
1160set pf8 imm b
1161/* goes to trace set a */
1162set pf1 imm tr goto a
1163/* goes to trace set b */
1164set pf2 imm tr goto b
1165/* goes to trace set c */
1166set pf3 imm tr goto c
1167
1168
1169
1170Instruction Tracing
1171-------------------
1172Setting a simple breakpoint
1173TR I PSWA <address>
1174To debug a particular function try
1175TR I R <function address range>
1176TR I on its own will single step.
1177TR I DATA <MNEMONIC> <OPTIONAL RANGE> will trace for particular mnemonics
1178e.g.
1179TR I DATA 4D R 0197BC.4000
1180will trace for BAS'es ( opcode 4D ) in the range 0197BC.4000
1181if you were inclined you could add traces for all branch instructions &
1182suffix them with the run prefix so you would have a backtrace on screen
1183when a program crashes.
1184TR BR <INTO OR FROM> will trace branches into or out of an address.
1185e.g.
1186TR BR INTO 0 is often quite useful if a program is getting awkward & deciding
1187to branch to 0 & crashing as this will stop at the address before in jumps to 0.
1188TR I R <address range> RUN cmd d g
1189single steps a range of addresses but stays running &
1190displays the gprs on each step.
1191
1192
1193
1194Displaying & modifying Registers
1195--------------------------------
1196D G will display all the gprs
1197Adding a extra G to all the commands is necessary to access the full 64 bit
1198content in VM on z/Architecture obviously this isn't required for access registers
1199as these are still 32 bit.
1200e.g. DGG instead of DG
1201D X will display all the control registers
1202D AR will display all the access registers
1203D AR4-7 will display access registers 4 to 7
1204CPU ALL D G will display the GRPS of all CPUS in the configuration
1205D PSW will display the current PSW
1206st PSW 2000 will put the value 2000 into the PSW &
1207cause crash your machine.
1208D PREFIX displays the prefix offset
1209
1210
1211Displaying Memory
1212-----------------
1213To display memory mapped using the current PSW's mapping try
1214D <range>
1215To make VM display a message each time it hits a particular address & continue try
1216D I<range> will disassemble/display a range of instructions.
1217ST addr 32 bit word will store a 32 bit aligned address
1218D T<range> will display the EBCDIC in an address ( if you are that way inclined )
1219D R<range> will display real addresses ( without DAT ) but with prefixing.
1220There are other complex options to display if you need to get at say home space
1221but are in primary space the easiest thing to do is to temporarily
1222modify the PSW to the other addressing mode, display the stuff & then
1223restore it.
1224
1225
1226
1227Hints
1228-----
1229If you want to issue a debugger command without halting your virtual machine with the
1230PA1 key try prefixing the command with #CP e.g.
1231#cp tr i pswa 2000
1232also suffixing most debugger commands with RUN will cause them not
1233to stop just display the mnemonic at the current instruction on the console.
1234If you have several breakpoints you want to put into your program &
1235you get fed up of cross referencing with System.map
1236you can do the following trick for several symbols.
1237grep do_signal System.map
1238which emits the following among other things
12390001f4e0 T do_signal
1240now you can do
1241
1242TR I PSWA 0001f4e0 cmd msg * do_signal
1243This sends a message to your own console each time do_signal is entered.
1244( As an aside I wrote a perl script once which automatically generated a REXX
1245script with breakpoints on every kernel procedure, this isn't a good idea
1246because there are thousands of these routines & VM can only set 255 breakpoints
1247at a time so you nearly had to spend as long pruning the file down as you would
1248entering the msg's by hand ),however, the trick might be useful for a single object file.
1249On linux'es 3270 emulator x3270 there is a very useful option under the file ment
1250Save Screens In File this is very good of keeping a copy of traces.
1251
1252From CMS help <command name> will give you online help on a particular command.
1253e.g.
1254HELP DISPLAY
1255
1256Also CP has a file called profile.exec which automatically gets called
1257on startup of CMS ( like autoexec.bat ), keeping on a DOS analogy session
1258CP has a feature similar to doskey, it may be useful for you to
1259use profile.exec to define some keystrokes.
1260e.g.
1261SET PF9 IMM B
1262This does a single step in VM on pressing F8.
1263SET PF10 ^
1264This sets up the ^ key.
1265which can be used for ^c (ctrl-c),^z (ctrl-z) which can't be typed directly into some 3270 consoles.
1266SET PF11 ^-
1267This types the starting keystrokes for a sysrq see SysRq below.
1268SET PF12 RETRIEVE
1269This retrieves command history on pressing F12.
1270
1271
1272Sometimes in VM the display is set up to scroll automatically this
1273can be very annoying if there are messages you wish to look at
1274to stop this do
1275TERM MORE 255 255
1276This will nearly stop automatic screen updates, however it will
1277cause a denial of service if lots of messages go to the 3270 console,
1278so it would be foolish to use this as the default on a production machine.
1279
1280
1281Tracing particular processes
1282----------------------------
1283The kernel's text segment is intentionally at an address in memory that it will
1284very seldom collide with text segments of user programs ( thanks Martin ),
1285this simplifies debugging the kernel.
1286However it is quite common for user processes to have addresses which collide
1287this can make debugging a particular process under VM painful under normal
1288circumstances as the process may change when doing a
1289TR I R <address range>.
1290Thankfully after reading VM's online help I figured out how to debug
1291I particular process.
1292
1293Your first problem is to find the STD ( segment table designation )
1294of the program you wish to debug.
1295There are several ways you can do this here are a few
12961) objdump --syms <program to be debugged> | grep main
1297To get the address of main in the program.
1298tr i pswa <address of main>
1299Start the program, if VM drops to CP on what looks like the entry
1300point of the main function this is most likely the process you wish to debug.
1301Now do a D X13 or D XG13 on z/Architecture.
1302On 31 bit the STD is bits 1-19 ( the STO segment table origin )
1303& 25-31 ( the STL segment table length ) of CR13.
1304now type
1305TR I R STD <CR13's value> 0.7fffffff
1306e.g.
1307TR I R STD 8F32E1FF 0.7fffffff
1308Another very useful variation is
1309TR STORE INTO STD <CR13's value> <address range>
1310for finding out when a particular variable changes.
1311
1312An alternative way of finding the STD of a currently running process
1313is to do the following, ( this method is more complex but
1314could be quite convient if you aren't updating the kernel much &
1315so your kernel structures will stay constant for a reasonable period of
1316time ).
1317
1318grep task /proc/<pid>/status
1319from this you should see something like
1320task: 0f160000 ksp: 0f161de8 pt_regs: 0f161f68
1321This now gives you a pointer to the task structure.
1322Now make CC:="s390-gcc -g" kernel/sched.s
1323To get the task_struct stabinfo.
1324( task_struct is defined in include/linux/sched.h ).
1325Now we want to look at
1326task->active_mm->pgd
1327on my machine the active_mm in the task structure stab is
1328active_mm:(4,12),672,32
1329its offset is 672/8=84=0x54
1330the pgd member in the mm_struct stab is
1331pgd:(4,6)=*(29,5),96,32
1332so its offset is 96/8=12=0xc
1333
1334so we'll
1335hexdump -s 0xf160054 /dev/mem | more
1336i.e. task_struct+active_mm offset
1337to look at the active_mm member
1338f160054 0fee cc60 0019 e334 0000 0000 0000 0011
1339hexdump -s 0x0feecc6c /dev/mem | more
1340i.e. active_mm+pgd offset
1341feecc6c 0f2c 0000 0000 0001 0000 0001 0000 0010
1342we get something like
1343now do
1344TR I R STD <pgd|0x7f> 0.7fffffff
1345i.e. the 0x7f is added because the pgd only
1346gives the page table origin & we need to set the low bits
1347to the maximum possible segment table length.
1348TR I R STD 0f2c007f 0.7fffffff
1349on z/Architecture you'll probably need to do
1350TR I R STD <pgd|0x7> 0.ffffffffffffffff
1351to set the TableType to 0x1 & the Table length to 3.
1352
1353
1354
1355Tracing Program Exceptions
1356--------------------------
1357If you get a crash which says something like
1358illegal operation or specification exception followed by a register dump
1359You can restart linux & trace these using the tr prog <range or value> trace option.
1360
1361
1362
1363The most common ones you will normally be tracing for is
13641=operation exception
13652=privileged operation exception
13664=protection exception
13675=addressing exception
13686=specification exception
136910=segment translation exception
137011=page translation exception
1371
1372The full list of these is on page 22 of the current s/390 Reference Summary.
1373e.g.
1374tr prog 10 will trace segment translation exceptions.
1375tr prog on its own will trace all program interruption codes.
1376
1377Trace Sets
1378----------
1379On starting VM you are initially in the INITIAL trace set.
1380You can do a Q TR to verify this.
1381If you have a complex tracing situation where you wish to wait for instance
1382till a driver is open before you start tracing IO, but know in your
1383heart that you are going to have to make several runs through the code till you
1384have a clue whats going on.
1385
1386What you can do is
1387TR I PSWA <Driver open address>
1388hit b to continue till breakpoint
1389reach the breakpoint
1390now do your
1391TR GOTO B
1392TR IO 7c08-7c09 inst int run
1393or whatever the IO channels you wish to trace are & hit b
1394
1395To got back to the initial trace set do
1396TR GOTO INITIAL
1397& the TR I PSWA <Driver open address> will be the only active breakpoint again.
1398
1399
1400Tracing linux syscalls under VM
1401-------------------------------
1402Syscalls are implemented on Linux for S390 by the Supervisor call instruction (SVC) there 256
1403possibilities of these as the instruction is made up of a 0xA opcode & the second byte being
1404the syscall number. They are traced using the simple command.
1405TR SVC <Optional value or range>
1406the syscalls are defined in linux/include/asm-s390/unistd.h
1407e.g. to trace all file opens just do
1408TR SVC 5 ( as this is the syscall number of open )
1409
1410
1411SMP Specific commands
1412---------------------
1413To find out how many cpus you have
1414Q CPUS displays all the CPU's available to your virtual machine
1415To find the cpu that the current cpu VM debugger commands are being directed at do
1416Q CPU to change the current cpu cpu VM debugger commands are being directed at do
1417CPU <desired cpu no>
1418
1419On a SMP guest issue a command to all CPUs try prefixing the command with cpu all.
1420To issue a command to a particular cpu try cpu <cpu number> e.g.
1421CPU 01 TR I R 2000.3000
1422If you are running on a guest with several cpus & you have a IO related problem
1423& cannot follow the flow of code but you know it isnt smp related.
1424from the bash prompt issue
1425shutdown -h now or halt.
1426do a Q CPUS to find out how many cpus you have
1427detach each one of them from cp except cpu 0
1428by issuing a
1429DETACH CPU 01-(number of cpus in configuration)
1430& boot linux again.
1431TR SIGP will trace inter processor signal processor instructions.
1432DEFINE CPU 01-(number in configuration)
1433will get your guests cpus back.
1434
1435
1436Help for displaying ascii textstrings
1437-------------------------------------
1438On the very latest VM Nucleus'es VM can now display ascii
1439( thanks Neale for the hint ) by doing
1440D TX<lowaddr>.<len>
1441e.g.
1442D TX0.100
1443
1444Alternatively
1445=============
1446Under older VM debuggers ( I love EBDIC too ) you can use this little program I wrote which
1447will convert a command line of hex digits to ascii text which can be compiled under linux &
1448you can copy the hex digits from your x3270 terminal to your xterm if you are debugging
1449from a linuxbox.
1450
1451This is quite useful when looking at a parameter passed in as a text string
1452under VM ( unless you are good at decoding ASCII in your head ).
1453
1454e.g. consider tracing an open syscall
1455TR SVC 5
1456We have stopped at a breakpoint
1457000151B0' SVC 0A05 -> 0001909A' CC 0
1458
1459D 20.8 to check the SVC old psw in the prefix area & see was it from userspace
1460( for the layout of the prefix area consult P18 of the s/390 390 Reference Summary
1461if you have it available ).
1462V00000020 070C2000 800151B2
1463The problem state bit wasn't set & it's also too early in the boot sequence
1464for it to be a userspace SVC if it was we would have to temporarily switch the
1465psw to user space addressing so we could get at the first parameter of the open in
1466gpr2.
1467Next do a
1468D G2
1469GPR 2 = 00014CB4
1470Now display what gpr2 is pointing to
1471D 00014CB4.20
1472V00014CB4 2F646576 2F636F6E 736F6C65 00001BF5
1473V00014CC4 FC00014C B4001001 E0001000 B8070707
1474Now copy the text till the first 00 hex ( which is the end of the string
1475to an xterm & do hex2ascii on it.
1476hex2ascii 2F646576 2F636F6E 736F6C65 00
1477outputs
1478Decoded Hex:=/ d e v / c o n s o l e 0x00
1479We were opening the console device,
1480
1481You can compile the code below yourself for practice :-),
1482/*
1483 * hex2ascii.c
1484 * a useful little tool for converting a hexadecimal command line to ascii
1485 *
1486 * Author(s): Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
1487 * (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation.
1488 */
1489#include <stdio.h>
1490
1491int main(int argc,char *argv[])
1492{
1493 int cnt1,cnt2,len,toggle=0;
1494 int startcnt=1;
1495 unsigned char c,hex;
1496
1497 if(argc>1&&(strcmp(argv[1],"-a")==0))
1498 startcnt=2;
1499 printf("Decoded Hex:=");
1500 for(cnt1=startcnt;cnt1<argc;cnt1++)
1501 {
1502 len=strlen(argv[cnt1]);
1503 for(cnt2=0;cnt2<len;cnt2++)
1504 {
1505 c=argv[cnt1][cnt2];
1506 if(c>='0'&&c<='9')
1507 c=c-'0';
1508 if(c>='A'&&c<='F')
1509 c=c-'A'+10;
1510 if(c>='a'&&c<='f')
1511 c=c-'a'+10;
1512 switch(toggle)
1513 {
1514 case 0:
1515 hex=c<<4;
1516 toggle=1;
1517 break;
1518 case 1:
1519 hex+=c;
1520 if(hex<32||hex>127)
1521 {
1522 if(startcnt==1)
1523 printf("0x%02X ",(int)hex);
1524 else
1525 printf(".");
1526 }
1527 else
1528 {
1529 printf("%c",hex);
1530 if(startcnt==1)
1531 printf(" ");
1532 }
1533 toggle=0;
1534 break;
1535 }
1536 }
1537 }
1538 printf("\n");
1539}
1540
1541
1542
1543
1544Stack tracing under VM
1545----------------------
1546A basic backtrace
1547-----------------
1548
1549Here are the tricks I use 9 out of 10 times it works pretty well,
1550
1551When your backchain reaches a dead end
1552--------------------------------------
1553This can happen when an exception happens in the kernel & the kernel is entered twice
1554if you reach the NULL pointer at the end of the back chain you should be
1555able to sniff further back if you follow the following tricks.
15561) A kernel address should be easy to recognise since it is in
1557primary space & the problem state bit isn't set & also
1558The Hi bit of the address is set.
15592) Another backchain should also be easy to recognise since it is an
1560address pointing to another address approximately 100 bytes or 0x70 hex
1561behind the current stackpointer.
1562
1563
1564Here is some practice.
1565boot the kernel & hit PA1 at some random time
1566d g to display the gprs, this should display something like
1567GPR 0 = 00000001 00156018 0014359C 00000000
1568GPR 4 = 00000001 001B8888 000003E0 00000000
1569GPR 8 = 00100080 00100084 00000000 000FE000
1570GPR 12 = 00010400 8001B2DC 8001B36A 000FFED8
1571Note that GPR14 is a return address but as we are real men we are going to
1572trace the stack.
1573display 0x40 bytes after the stack pointer.
1574
1575V000FFED8 000FFF38 8001B838 80014C8E 000FFF38
1576V000FFEE8 00000000 00000000 000003E0 00000000
1577V000FFEF8 00100080 00100084 00000000 000FE000
1578V000FFF08 00010400 8001B2DC 8001B36A 000FFED8
1579
1580
1581Ah now look at whats in sp+56 (sp+0x38) this is 8001B36A our saved r14 if
1582you look above at our stackframe & also agrees with GPR14.
1583
1584now backchain
1585d 000FFF38.40
1586we now are taking the contents of SP to get our first backchain.
1587
1588V000FFF38 000FFFA0 00000000 00014995 00147094
1589V000FFF48 00147090 001470A0 000003E0 00000000
1590V000FFF58 00100080 00100084 00000000 001BF1D0
1591V000FFF68 00010400 800149BA 80014CA6 000FFF38
1592
1593This displays a 2nd return address of 80014CA6
1594
1595now do d 000FFFA0.40 for our 3rd backchain
1596
1597V000FFFA0 04B52002 0001107F 00000000 00000000
1598V000FFFB0 00000000 00000000 FF000000 0001107F
1599V000FFFC0 00000000 00000000 00000000 00000000
1600V000FFFD0 00010400 80010802 8001085A 000FFFA0
1601
1602
1603our 3rd return address is 8001085A
1604
1605as the 04B52002 looks suspiciously like rubbish it is fair to assume that the kernel entry routines
1606for the sake of optimisation dont set up a backchain.
1607
1608now look at System.map to see if the addresses make any sense.
1609
1610grep -i 0001b3 System.map
1611outputs among other things
16120001b304 T cpu_idle
1613so 8001B36A
1614is cpu_idle+0x66 ( quiet the cpu is asleep, don't wake it )
1615
1616
1617grep -i 00014 System.map
1618produces among other things
161900014a78 T start_kernel
1620so 0014CA6 is start_kernel+some hex number I can't add in my head.
1621
1622grep -i 00108 System.map
1623this produces
162400010800 T _stext
1625so 8001085A is _stext+0x5a
1626
1627Congrats you've done your first backchain.
1628
1629
1630
1631s/390 & z/Architecture IO Overview
1632==================================
1633
1634I am not going to give a course in 390 IO architecture as this would take me quite a
1635while & I'm no expert. Instead I'll give a 390 IO architecture summary for Dummies if you have
1636the s/390 principles of operation available read this instead. If nothing else you may find a few
1637useful keywords in here & be able to use them on a web search engine like altavista to find
1638more useful information.
1639
1640Unlike other bus architectures modern 390 systems do their IO using mostly
1641fibre optics & devices such as tapes & disks can be shared between several mainframes,
1642also S390 can support upto 65536 devices while a high end PC based system might be choking
1643with around 64. Here is some of the common IO terminology
1644
1645Subchannel:
1646This is the logical number most IO commands use to talk to an IO device there can be upto
16470x10000 (65536) of these in a configuration typically there is a few hundred. Under VM
1648for simplicity they are allocated contiguously, however on the native hardware they are not
1649they typically stay consistent between boots provided no new hardware is inserted or removed.
1650Under Linux for 390 we use these as IRQ's & also when issuing an IO command (CLEAR SUBCHANNEL,
1651HALT SUBCHANNEL,MODIFY SUBCHANNEL,RESUME SUBCHANNEL,START SUBCHANNEL,STORE SUBCHANNEL &
1652TEST SUBCHANNEL ) we use this as the ID of the device we wish to talk to, the most
1653important of these instructions are START SUBCHANNEL ( to start IO ), TEST SUBCHANNEL ( to check
1654whether the IO completed successfully ), & HALT SUBCHANNEL ( to kill IO ), a subchannel
1655can have up to 8 channel paths to a device this offers redunancy if one is not available.
1656
1657
1658Device Number:
1659This number remains static & Is closely tied to the hardware, there are 65536 of these
1660also they are made up of a CHPID ( Channel Path ID, the most significant 8 bits )
1661& another lsb 8 bits. These remain static even if more devices are inserted or removed
1662from the hardware, there is a 1 to 1 mapping between Subchannels & Device Numbers provided
1663devices arent inserted or removed.
1664
1665Channel Control Words:
1666CCWS are linked lists of instructions initially pointed to by an operation request block (ORB),
1667which is initially given to Start Subchannel (SSCH) command along with the subchannel number
1668for the IO subsystem to process while the CPU continues executing normal code.
1669These come in two flavours, Format 0 ( 24 bit for backward )
1670compatibility & Format 1 ( 31 bit ). These are typically used to issue read & write
1671( & many other instructions ) they consist of a length field & an absolute address field.
1672For each IO typically get 1 or 2 interrupts one for channel end ( primary status ) when the
1673channel is idle & the second for device end ( secondary status ) sometimes you get both
1674concurrently, you check how the IO went on by issuing a TEST SUBCHANNEL at each interrupt,
1675from which you receive an Interruption response block (IRB). If you get channel & device end
1676status in the IRB without channel checks etc. your IO probably went okay. If you didn't you
1677probably need a doctorto examine the IRB & extended status word etc.
1678If an error occurs more sophistocated control units have a facitity known as
1679concurrent sense this means that if an error occurs Extended sense information will
1680be presented in the Extended status word in the IRB if not you have to issue a
1681subsequent SENSE CCW command after the test subchannel.
1682
1683
1684TPI( Test pending interrupt) can also be used for polled IO but in multitasking multiprocessor
1685systems it isn't recommended except for checking special cases ( i.e. non looping checks for
1686pending IO etc. ).
1687
1688Store Subchannel & Modify Subchannel can be used to examine & modify operating characteristics
1689of a subchannel ( e.g. channel paths ).
1690
1691Other IO related Terms:
1692Sysplex: S390's Clustering Technology
1693QDIO: S390's new high speed IO architecture to support devices such as gigabit ethernet,
1694this architecture is also designed to be forward compatible with up & coming 64 bit machines.
1695
1696
1697General Concepts
1698
1699Input Output Processors (IOP's) are responsible for communicating between
1700the mainframe CPU's & the channel & relieve the mainframe CPU's from the
1701burden of communicating with IO devices directly, this allows the CPU's to
1702concentrate on data processing.
1703
1704IOP's can use one or more links ( known as channel paths ) to talk to each
1705IO device. It first checks for path availability & chooses an available one,
1706then starts ( & sometimes terminates IO ).
1707There are two types of channel path ESCON & the Paralell IO interface.
1708
1709IO devices are attached to control units, control units provide the
1710logic to interface the channel paths & channel path IO protocols to
1711the IO devices, they can be integrated with the devices or housed separately
1712& often talk to several similar devices ( typical examples would be raid
1713controllers or a control unit which connects to 1000 3270 terminals ).
1714
1715
1716 +---------------------------------------------------------------+
1717 | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+ |
1718 | | CPU | | CPU | | CPU | | CPU | | Main | | Expanded | |
1719 | | | | | | | | | | Memory | | Storage | |
1720 | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+ |
1721 |---------------------------------------------------------------+
1722 | IOP | IOP | IOP |
1723 |---------------------------------------------------------------
1724 | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C |
1725 ----------------------------------------------------------------
1726 || ||
1727 || Bus & Tag Channel Path || ESCON
1728 || ====================== || Channel
1729 || || || || Path
1730 +----------+ +----------+ +----------+
1731 | | | | | |
1732 | CU | | CU | | CU |
1733 | | | | | |
1734 +----------+ +----------+ +----------+
1735 | | | | |
1736+----------+ +----------+ +----------+ +----------+ +----------+
1737|I/O Device| |I/O Device| |I/O Device| |I/O Device| |I/O Device|
1738+----------+ +----------+ +----------+ +----------+ +----------+
1739 CPU = Central Processing Unit
1740 C = Channel
1741 IOP = IP Processor
1742 CU = Control Unit
1743
1744The 390 IO systems come in 2 flavours the current 390 machines support both
1745
1746The Older 360 & 370 Interface,sometimes called the paralell I/O interface,
1747sometimes called Bus-and Tag & sometimes Original Equipment Manufacturers
1748Interface (OEMI).
1749
1750This byte wide paralell channel path/bus has parity & data on the "Bus" cable
1751& control lines on the "Tag" cable. These can operate in byte multiplex mode for
1752sharing between several slow devices or burst mode & monopolize the channel for the
1753whole burst. Upto 256 devices can be addressed on one of these cables. These cables are
1754about one inch in diameter. The maximum unextended length supported by these cables is
1755125 Meters but this can be extended up to 2km with a fibre optic channel extended
1756such as a 3044. The maximum burst speed supported is 4.5 megabytes per second however
1757some really old processors support only transfer rates of 3.0, 2.0 & 1.0 MB/sec.
1758One of these paths can be daisy chained to up to 8 control units.
1759
1760
1761ESCON if fibre optic it is also called FICON
1762Was introduced by IBM in 1990. Has 2 fibre optic cables & uses either leds or lasers
1763for communication at a signaling rate of upto 200 megabits/sec. As 10bits are transferred
1764for every 8 bits info this drops to 160 megabits/sec & to 18.6 Megabytes/sec once
1765control info & CRC are added. ESCON only operates in burst mode.
1766
1767ESCONs typical max cable length is 3km for the led version & 20km for the laser version
1768known as XDF ( extended distance facility ). This can be further extended by using an
1769ESCON director which triples the above mentioned ranges. Unlike Bus & Tag as ESCON is
1770serial it uses a packet switching architecture the standard Bus & Tag control protocol
1771is however present within the packets. Upto 256 devices can be attached to each control
1772unit that uses one of these interfaces.
1773
1774Common 390 Devices include:
1775Network adapters typically OSA2,3172's,2116's & OSA-E gigabit ethernet adapters,
1776Consoles 3270 & 3215 ( a teletype emulated under linux for a line mode console ).
1777DASD's direct access storage devices ( otherwise known as hard disks ).
1778Tape Drives.
1779CTC ( Channel to Channel Adapters ),
1780ESCON or Paralell Cables used as a very high speed serial link
1781between 2 machines. We use 2 cables under linux to do a bi-directional serial link.
1782
1783
1784Debugging IO on s/390 & z/Architecture under VM
1785===============================================
1786
1787Now we are ready to go on with IO tracing commands under VM
1788
1789A few self explanatory queries:
1790Q OSA
1791Q CTC
1792Q DISK ( This command is CMS specific )
1793Q DASD
1794
1795
1796
1797
1798
1799
1800Q OSA on my machine returns
1801OSA 7C08 ON OSA 7C08 SUBCHANNEL = 0000
1802OSA 7C09 ON OSA 7C09 SUBCHANNEL = 0001
1803OSA 7C14 ON OSA 7C14 SUBCHANNEL = 0002
1804OSA 7C15 ON OSA 7C15 SUBCHANNEL = 0003
1805
1806If you have a guest with certain priviliges you may be able to see devices
1807which don't belong to you to avoid this do add the option V.
1808e.g.
1809Q V OSA
1810
1811Now using the device numbers returned by this command we will
1812Trace the io starting up on the first device 7c08 & 7c09
1813In our simplest case we can trace the
1814start subchannels
1815like TR SSCH 7C08-7C09
1816or the halt subchannels
1817or TR HSCH 7C08-7C09
1818MSCH's ,STSCH's I think you can guess the rest
1819
1820Ingo's favourite trick is tracing all the IO's & CCWS & spooling them into the reader of another
1821VM guest so he can ftp the logfile back to his own machine.I'll do a small bit of this & give you
1822 a look at the output.
1823
18241) Spool stdout to VM reader
1825SP PRT TO (another vm guest ) or * for the local vm guest
18262) Fill the reader with the trace
1827TR IO 7c08-7c09 INST INT CCW PRT RUN
18283) Start up linux
1829i 00c
18304) Finish the trace
1831TR END
18325) close the reader
1833C PRT
18346) list reader contents
1835RDRLIST
18367) copy it to linux4's minidisk
1837RECEIVE / LOG TXT A1 ( replace
18388)
1839filel & press F11 to look at it
1840You should see someting like.
1841
184200020942' SSCH B2334000 0048813C CC 0 SCH 0000 DEV 7C08
1843 CPA 000FFDF0 PARM 00E2C9C4 KEY 0 FPI C0 LPM 80
1844 CCW 000FFDF0 E4200100 00487FE8 0000 E4240100 ........
1845 IDAL 43D8AFE8
1846 IDAL 0FB76000
184700020B0A' I/O DEV 7C08 -> 000197BC' SCH 0000 PARM 00E2C9C4
184800021628' TSCH B2354000 >> 00488164 CC 0 SCH 0000 DEV 7C08
1849 CCWA 000FFDF8 DEV STS 0C SCH STS 00 CNT 00EC
1850 KEY 0 FPI C0 CC 0 CTLS 4007
185100022238' STSCH B2344000 >> 00488108 CC 0 SCH 0000 DEV 7C08
1852
1853If you don't like messing up your readed ( because you possibly booted from it )
1854you can alternatively spool it to another readers guest.
1855
1856
1857Other common VM device related commands
1858---------------------------------------------
1859These commands are listed only because they have
1860been of use to me in the past & may be of use to
1861you too. For more complete info on each of the commands
1862use type HELP <command> from CMS.
1863detaching devices
1864DET <devno range>
1865ATT <devno range> <guest>
1866attach a device to guest * for your own guest
1867READY <devno> cause VM to issue a fake interrupt.
1868
1869The VARY command is normally only available to VM administrators.
1870VARY ON PATH <path> TO <devno range>
1871VARY OFF PATH <PATH> FROM <devno range>
1872This is used to switch on or off channel paths to devices.
1873
1874Q CHPID <channel path ID>
1875This displays state of devices using this channel path
1876D SCHIB <subchannel>
1877This displays the subchannel information SCHIB block for the device.
1878this I believe is also only available to administrators.
1879DEFINE CTC <devno>
1880defines a virtual CTC channel to channel connection
18812 need to be defined on each guest for the CTC driver to use.
1882COUPLE devno userid remote devno
1883Joins a local virtual device to a remote virtual device
1884( commonly used for the CTC driver ).
1885
1886Building a VM ramdisk under CMS which linux can use
1887def vfb-<blocksize> <subchannel> <number blocks>
1888blocksize is commonly 4096 for linux.
1889Formatting it
1890format <subchannel> <driver letter e.g. x> (blksize <blocksize>
1891
1892Sharing a disk between multiple guests
1893LINK userid devno1 devno2 mode password
1894
1895
1896
1897GDB on S390
1898===========
1899N.B. if compiling for debugging gdb works better without optimisation
1900( see Compiling programs for debugging )
1901
1902invocation
1903----------
1904gdb <victim program> <optional corefile>
1905
1906Online help
1907-----------
1908help: gives help on commands
1909e.g.
1910help
1911help display
1912Note gdb's online help is very good use it.
1913
1914
1915Assembly
1916--------
1917info registers: displays registers other than floating point.
1918info all-registers: displays floating points as well.
1919disassemble: dissassembles
1920e.g.
1921disassemble without parameters will disassemble the current function
1922disassemble $pc $pc+10
1923
1924Viewing & modifying variables
1925-----------------------------
1926print or p: displays variable or register
1927e.g. p/x $sp will display the stack pointer
1928
1929display: prints variable or register each time program stops
1930e.g.
1931display/x $pc will display the program counter
1932display argc
1933
1934undisplay : undo's display's
1935
1936info breakpoints: shows all current breakpoints
1937
1938info stack: shows stack back trace ( if this dosent work too well, I'll show you the
1939stacktrace by hand below ).
1940
1941info locals: displays local variables.
1942
1943info args: display current procedure arguments.
1944
1945set args: will set argc & argv each time the victim program is invoked.
1946
1947set <variable>=value
1948set argc=100
1949set $pc=0
1950
1951
1952
1953Modifying execution
1954-------------------
1955step: steps n lines of sourcecode
1956step steps 1 line.
1957step 100 steps 100 lines of code.
1958
1959next: like step except this will not step into subroutines
1960
1961stepi: steps a single machine code instruction.
1962e.g. stepi 100
1963
1964nexti: steps a single machine code instruction but will not step into subroutines.
1965
1966finish: will run until exit of the current routine
1967
1968run: (re)starts a program
1969
1970cont: continues a program
1971
1972quit: exits gdb.
1973
1974
1975breakpoints
1976------------
1977
1978break
1979sets a breakpoint
1980e.g.
1981
1982break main
1983
1984break *$pc
1985
1986break *0x400618
1987
1988heres a really useful one for large programs
1989rbr
1990Set a breakpoint for all functions matching REGEXP
1991e.g.
1992rbr 390
1993will set a breakpoint with all functions with 390 in their name.
1994
1995info breakpoints
1996lists all breakpoints
1997
1998delete: delete breakpoint by number or delete them all
1999e.g.
2000delete 1 will delete the first breakpoint
2001delete will delete them all
2002
2003watch: This will set a watchpoint ( usually hardware assisted ),
2004This will watch a variable till it changes
2005e.g.
2006watch cnt, will watch the variable cnt till it changes.
2007As an aside unfortunately gdb's, architecture independent watchpoint code
2008is inconsistent & not very good, watchpoints usually work but not always.
2009
2010info watchpoints: Display currently active watchpoints
2011
2012condition: ( another useful one )
2013Specify breakpoint number N to break only if COND is true.
2014Usage is `condition N COND', where N is an integer and COND is an
2015expression to be evaluated whenever breakpoint N is reached.
2016
2017
2018
2019User defined functions/macros
2020-----------------------------
2021define: ( Note this is very very useful,simple & powerful )
2022usage define <name> <list of commands> end
2023
2024examples which you should consider putting into .gdbinit in your home directory
2025define d
2026stepi
2027disassemble $pc $pc+10
2028end
2029
2030define e
2031nexti
2032disassemble $pc $pc+10
2033end
2034
2035
2036Other hard to classify stuff
2037----------------------------
2038signal n:
2039sends the victim program a signal.
2040e.g. signal 3 will send a SIGQUIT.
2041
2042info signals:
2043what gdb does when the victim receives certain signals.
2044
2045list:
2046e.g.
2047list lists current function source
2048list 1,10 list first 10 lines of curret file.
2049list test.c:1,10
2050
2051
2052directory:
2053Adds directories to be searched for source if gdb cannot find the source.
2054(note it is a bit sensititive about slashes )
2055e.g. To add the root of the filesystem to the searchpath do
2056directory //
2057
2058
2059call <function>
2060This calls a function in the victim program, this is pretty powerful
2061e.g.
2062(gdb) call printf("hello world")
2063outputs:
2064$1 = 11
2065
2066You might now be thinking that the line above didn't work, something extra had to be done.
2067(gdb) call fflush(stdout)
2068hello world$2 = 0
2069As an aside the debugger also calls malloc & free under the hood
2070to make space for the "hello world" string.
2071
2072
2073
2074hints
2075-----
20761) command completion works just like bash
2077( if you are a bad typist like me this really helps )
2078e.g. hit br <TAB> & cursor up & down :-).
2079
20802) if you have a debugging problem that takes a few steps to recreate
2081put the steps into a file called .gdbinit in your current working directory
2082if you have defined a few extra useful user defined commands put these in
2083your home directory & they will be read each time gdb is launched.
2084
2085A typical .gdbinit file might be.
2086break main
2087run
2088break runtime_exception
2089cont
2090
2091
2092stack chaining in gdb by hand
2093-----------------------------
2094This is done using a the same trick described for VM
2095p/x (*($sp+56))&0x7fffffff get the first backchain.
2096
2097For z/Architecture
2098Replace 56 with 112 & ignore the &0x7fffffff
2099in the macros below & do nasty casts to longs like the following
2100as gdb unfortunately deals with printed arguments as ints which
2101messes up everything.
2102i.e. here is a 3rd backchain dereference
2103p/x *(long *)(***(long ***)$sp+112)
2104
2105
2106this outputs
2107$5 = 0x528f18
2108on my machine.
2109Now you can use
2110info symbol (*($sp+56))&0x7fffffff
2111you might see something like.
2112rl_getc + 36 in section .text telling you what is located at address 0x528f18
2113Now do.
2114p/x (*(*$sp+56))&0x7fffffff
2115This outputs
2116$6 = 0x528ed0
2117Now do.
2118info symbol (*(*$sp+56))&0x7fffffff
2119rl_read_key + 180 in section .text
2120now do
2121p/x (*(**$sp+56))&0x7fffffff
2122& so on.
2123
2124Disassembling instructions without debug info
2125---------------------------------------------
2126gdb typically compains if there is a lack of debugging
2127symbols in the disassemble command with
2128"No function contains specified address." to get around
2129this do
2130x/<number lines to disassemble>xi <address>
2131e.g.
2132x/20xi 0x400730
2133
2134
2135
2136Note: Remember gdb has history just like bash you don't need to retype the
2137whole line just use the up & down arrows.
2138
2139
2140
2141For more info
2142-------------
2143From your linuxbox do
2144man gdb or info gdb.
2145
2146core dumps
2147----------
2148What a core dump ?,
2149A core dump is a file generated by the kernel ( if allowed ) which contains the registers,
2150& all active pages of the program which has crashed.
2151From this file gdb will allow you to look at the registers & stack trace & memory of the
2152program as if it just crashed on your system, it is usually called core & created in the
2153current working directory.
2154This is very useful in that a customer can mail a core dump to a technical support department
2155& the technical support department can reconstruct what happened.
2156Provided the have an identical copy of this program with debugging symbols compiled in &
2157the source base of this build is available.
2158In short it is far more useful than something like a crash log could ever hope to be.
2159
2160In theory all that is missing to restart a core dumped program is a kernel patch which
2161will do the following.
21621) Make a new kernel task structure
21632) Reload all the dumped pages back into the kernel's memory management structures.
21643) Do the required clock fixups
21654) Get all files & network connections for the process back into an identical state ( really difficult ).
21665) A few more difficult things I haven't thought of.
2167
2168
2169
2170Why have I never seen one ?.
2171Probably because you haven't used the command
2172ulimit -c unlimited in bash
2173to allow core dumps, now do
2174ulimit -a
2175to verify that the limit was accepted.
2176
2177A sample core dump
2178To create this I'm going to do
2179ulimit -c unlimited
2180gdb
2181to launch gdb (my victim app. ) now be bad & do the following from another
2182telnet/xterm session to the same machine
2183ps -aux | grep gdb
2184kill -SIGSEGV <gdb's pid>
2185or alternatively use killall -SIGSEGV gdb if you have the killall command.
2186Now look at the core dump.
2187./gdb ./gdb core
2188Displays the following
2189GNU gdb 4.18
2190Copyright 1998 Free Software Foundation, Inc.
2191GDB is free software, covered by the GNU General Public License, and you are
2192welcome to change it and/or distribute copies of it under certain conditions.
2193Type "show copying" to see the conditions.
2194There is absolutely no warranty for GDB. Type "show warranty" for details.
2195This GDB was configured as "s390-ibm-linux"...
2196Core was generated by `./gdb'.
2197Program terminated with signal 11, Segmentation fault.
2198Reading symbols from /usr/lib/libncurses.so.4...done.
2199Reading symbols from /lib/libm.so.6...done.
2200Reading symbols from /lib/libc.so.6...done.
2201Reading symbols from /lib/ld-linux.so.2...done.
2202#0 0x40126d1a in read () from /lib/libc.so.6
2203Setting up the environment for debugging gdb.
2204Breakpoint 1 at 0x4dc6f8: file utils.c, line 471.
2205Breakpoint 2 at 0x4d87a4: file top.c, line 2609.
2206(top-gdb) info stack
2207#0 0x40126d1a in read () from /lib/libc.so.6
2208#1 0x528f26 in rl_getc (stream=0x7ffffde8) at input.c:402
2209#2 0x528ed0 in rl_read_key () at input.c:381
2210#3 0x5167e6 in readline_internal_char () at readline.c:454
2211#4 0x5168ee in readline_internal_charloop () at readline.c:507
2212#5 0x51692c in readline_internal () at readline.c:521
2213#6 0x5164fe in readline (prompt=0x7ffff810 "\177ÿøx\177ÿ÷Ø\177ÿøxÀ")
2214 at readline.c:349
2215#7 0x4d7a8a in command_line_input (prrompt=0x564420 "(gdb) ", repeat=1,
2216 annotation_suffix=0x4d6b44 "prompt") at top.c:2091
2217#8 0x4d6cf0 in command_loop () at top.c:1345
2218#9 0x4e25bc in main (argc=1, argv=0x7ffffdf4) at main.c:635
2219
2220
2221LDD
2222===
2223This is a program which lists the shared libraries which a library needs,
2224Note you also get the relocations of the shared library text segments which
2225help when using objdump --source.
2226e.g.
2227 ldd ./gdb
2228outputs
2229libncurses.so.4 => /usr/lib/libncurses.so.4 (0x40018000)
2230libm.so.6 => /lib/libm.so.6 (0x4005e000)
2231libc.so.6 => /lib/libc.so.6 (0x40084000)
2232/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
2233
2234
2235Debugging shared libraries
2236==========================
2237Most programs use shared libraries, however it can be very painful
2238when you single step instruction into a function like printf for the
2239first time & you end up in functions like _dl_runtime_resolve this is
2240the ld.so doing lazy binding, lazy binding is a concept in ELF where
2241shared library functions are not loaded into memory unless they are
2242actually used, great for saving memory but a pain to debug.
2243To get around this either relink the program -static or exit gdb type
2244export LD_BIND_NOW=true this will stop lazy binding & restart the gdb'ing
2245the program in question.
2246
2247
2248
2249Debugging modules
2250=================
2251As modules are dynamically loaded into the kernel their address can be
2252anywhere to get around this use the -m option with insmod to emit a load
2253map which can be piped into a file if required.
2254
2255The proc file system
2256====================
2257What is it ?.
2258It is a filesystem created by the kernel with files which are created on demand
2259by the kernel if read, or can be used to modify kernel parameters,
2260it is a powerful concept.
2261
2262e.g.
2263
2264cat /proc/sys/net/ipv4/ip_forward
2265On my machine outputs
22660
2267telling me ip_forwarding is not on to switch it on I can do
2268echo 1 > /proc/sys/net/ipv4/ip_forward
2269cat it again
2270cat /proc/sys/net/ipv4/ip_forward
2271On my machine now outputs
22721
2273IP forwarding is on.
2274There is a lot of useful info in here best found by going in & having a look around,
2275so I'll take you through some entries I consider important.
2276
2277All the processes running on the machine have there own entry defined by
2278/proc/<pid>
2279So lets have a look at the init process
2280cd /proc/1
2281
2282cat cmdline
2283emits
2284init [2]
2285
2286cd /proc/1/fd
2287This contains numerical entries of all the open files,
2288some of these you can cat e.g. stdout (2)
2289
2290cat /proc/29/maps
2291on my machine emits
2292
229300400000-00478000 r-xp 00000000 5f:00 4103 /bin/bash
229400478000-0047e000 rw-p 00077000 5f:00 4103 /bin/bash
22950047e000-00492000 rwxp 00000000 00:00 0
229640000000-40015000 r-xp 00000000 5f:00 14382 /lib/ld-2.1.2.so
229740015000-40016000 rw-p 00014000 5f:00 14382 /lib/ld-2.1.2.so
229840016000-40017000 rwxp 00000000 00:00 0
229940017000-40018000 rw-p 00000000 00:00 0
230040018000-4001b000 r-xp 00000000 5f:00 14435 /lib/libtermcap.so.2.0.8
23014001b000-4001c000 rw-p 00002000 5f:00 14435 /lib/libtermcap.so.2.0.8
23024001c000-4010d000 r-xp 00000000 5f:00 14387 /lib/libc-2.1.2.so
23034010d000-40111000 rw-p 000f0000 5f:00 14387 /lib/libc-2.1.2.so
230440111000-40114000 rw-p 00000000 00:00 0
230540114000-4011e000 r-xp 00000000 5f:00 14408 /lib/libnss_files-2.1.2.so
23064011e000-4011f000 rw-p 00009000 5f:00 14408 /lib/libnss_files-2.1.2.so
23077fffd000-80000000 rwxp ffffe000 00:00 0
2308
2309
2310Showing us the shared libraries init uses where they are in memory
2311& memory access permissions for each virtual memory area.
2312
2313/proc/1/cwd is a softlink to the current working directory.
2314/proc/1/root is the root of the filesystem for this process.
2315
2316/proc/1/mem is the current running processes memory which you
2317can read & write to like a file.
2318strace uses this sometimes as it is a bit faster than the
2319rather inefficent ptrace interface for peeking at DATA.
2320
2321
2322cat status
2323
2324Name: init
2325State: S (sleeping)
2326Pid: 1
2327PPid: 0
2328Uid: 0 0 0 0
2329Gid: 0 0 0 0
2330Groups:
2331VmSize: 408 kB
2332VmLck: 0 kB
2333VmRSS: 208 kB
2334VmData: 24 kB
2335VmStk: 8 kB
2336VmExe: 368 kB
2337VmLib: 0 kB
2338SigPnd: 0000000000000000
2339SigBlk: 0000000000000000
2340SigIgn: 7fffffffd7f0d8fc
2341SigCgt: 00000000280b2603
2342CapInh: 00000000fffffeff
2343CapPrm: 00000000ffffffff
2344CapEff: 00000000fffffeff
2345
2346User PSW: 070de000 80414146
2347task: 004b6000 tss: 004b62d8 ksp: 004b7ca8 pt_regs: 004b7f68
2348User GPRS:
234900000400 00000000 0000000b 7ffffa90
235000000000 00000000 00000000 0045d9f4
23510045cafc 7ffffa90 7fffff18 0045cb08
235200010400 804039e8 80403af8 7ffff8b0
2353User ACRS:
235400000000 00000000 00000000 00000000
235500000001 00000000 00000000 00000000
235600000000 00000000 00000000 00000000
235700000000 00000000 00000000 00000000
2358Kernel BackChain CallChain BackChain CallChain
2359 004b7ca8 8002bd0c 004b7d18 8002b92c
2360 004b7db8 8005cd50 004b7e38 8005d12a
2361 004b7f08 80019114
2362Showing among other things memory usage & status of some signals &
2363the processes'es registers from the kernel task_structure
2364as well as a backchain which may be useful if a process crashes
2365in the kernel for some unknown reason.
2366
2367Some driver debugging techniques
2368================================
2369debug feature
2370-------------
2371Some of our drivers now support a "debug feature" in
2372/proc/s390dbf see s390dbf.txt in the linux/Documentation directory
2373for more info.
2374e.g.
2375to switch on the lcs "debug feature"
2376echo 5 > /proc/s390dbf/lcs/level
2377& then after the error occurred.
2378cat /proc/s390dbf/lcs/sprintf >/logfile
2379the logfile now contains some information which may help
2380tech support resolve a problem in the field.
2381
2382
2383
2384high level debugging network drivers
2385------------------------------------
2386ifconfig is a quite useful command
2387it gives the current state of network drivers.
2388
2389If you suspect your network device driver is dead
2390one way to check is type
2391ifconfig <network device>
2392e.g. tr0
2393You should see something like
2394tr0 Link encap:16/4 Mbps Token Ring (New) HWaddr 00:04:AC:20:8E:48
2395 inet addr:9.164.185.132 Bcast:9.164.191.255 Mask:255.255.224.0
2396 UP BROADCAST RUNNING MULTICAST MTU:2000 Metric:1
2397 RX packets:246134 errors:0 dropped:0 overruns:0 frame:0
2398 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
2399 collisions:0 txqueuelen:100
2400
2401if the device doesn't say up
2402try
2403/etc/rc.d/init.d/network start
2404( this starts the network stack & hopefully calls ifconfig tr0 up ).
2405ifconfig looks at the output of /proc/net/dev & presents it in a more presentable form
2406Now ping the device from a machine in the same subnet.
2407if the RX packets count & TX packets counts don't increment you probably
2408have problems.
2409next
2410cat /proc/net/arp
2411Do you see any hardware addresses in the cache if not you may have problems.
2412Next try
2413ping -c 5 <broadcast_addr> i.e. the Bcast field above in the output of
2414ifconfig. Do you see any replies from machines other than the local machine
2415if not you may have problems. also if the TX packets count in ifconfig
2416hasn't incremented either you have serious problems in your driver
2417(e.g. the txbusy field of the network device being stuck on )
2418or you may have multiple network devices connected.
2419
2420
2421chandev
2422-------
2423There is a new device layer for channel devices, some
2424drivers e.g. lcs are registered with this layer.
2425If the device uses the channel device layer you'll be
2426able to find what interrupts it uses & the current state
2427of the device.
2428See the manpage chandev.8 &type cat /proc/chandev for more info.
2429
2430
2431
2432Starting points for debugging scripting languages etc.
2433======================================================
2434
2435bash/sh
2436
2437bash -x <scriptname>
2438e.g. bash -x /usr/bin/bashbug
2439displays the following lines as it executes them.
2440+ MACHINE=i586
2441+ OS=linux-gnu
2442+ CC=gcc
2443+ CFLAGS= -DPROGRAM='bash' -DHOSTTYPE='i586' -DOSTYPE='linux-gnu' -DMACHTYPE='i586-pc-linux-gnu' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./lib -O2 -pipe
2444+ RELEASE=2.01
2445+ PATCHLEVEL=1
2446+ RELSTATUS=release
2447+ MACHTYPE=i586-pc-linux-gnu
2448
2449perl -d <scriptname> runs the perlscript in a fully intercative debugger
2450<like gdb>.
2451Type 'h' in the debugger for help.
2452
2453for debugging java type
2454jdb <filename> another fully interactive gdb style debugger.
2455& type ? in the debugger for help.
2456
2457
2458
2459Dumptool & Lcrash ( lkcd )
2460==========================
2461Michael Holzheu & others here at IBM have a fairly mature port of
2462SGI's lcrash tool which allows one to look at kernel structures in a
2463running kernel.
2464
2465It also complements a tool called dumptool which dumps all the kernel's
2466memory pages & registers to either a tape or a disk.
2467This can be used by tech support or an ambitious end user do
2468post mortem debugging of a machine like gdb core dumps.
2469
2470Going into how to use this tool in detail will be explained
2471in other documentation supplied by IBM with the patches & the
2472lcrash homepage http://oss.sgi.com/projects/lkcd/ & the lcrash manpage.
2473
2474How they work
2475-------------
2476Lcrash is a perfectly normal program,however, it requires 2
2477additional files, Kerntypes which is built using a patch to the
2478linux kernel sources in the linux root directory & the System.map.
2479
2480Kerntypes is an an objectfile whose sole purpose in life
2481is to provide stabs debug info to lcrash, to do this
2482Kerntypes is built from kerntypes.c which just includes the most commonly
2483referenced header files used when debugging, lcrash can then read the
2484.stabs section of this file.
2485
2486Debugging a live system it uses /dev/mem
2487alternatively for post mortem debugging it uses the data
2488collected by dumptool.
2489
2490
2491
2492SysRq
2493=====
2494This is now supported by linux for s/390 & z/Architecture.
2495To enable it do compile the kernel with
2496Kernel Hacking -> Magic SysRq Key Enabled
2497echo "1" > /proc/sys/kernel/sysrq
2498also type
2499echo "8" >/proc/sys/kernel/printk
2500To make printk output go to console.
2501On 390 all commands are prefixed with
2502^-
2503e.g.
2504^-t will show tasks.
2505^-? or some unknown command will display help.
2506The sysrq key reading is very picky ( I have to type the keys in an
2507 xterm session & paste them into the x3270 console )
2508& it may be wise to predefine the keys as described in the VM hints above
2509
2510This is particularly useful for syncing disks unmounting & rebooting
2511if the machine gets partially hung.
2512
2513Read Documentation/sysrq.txt for more info
2514
2515References:
2516===========
2517Enterprise Systems Architecture Reference Summary
2518Enterprise Systems Architecture Principles of Operation
2519Hartmut Penners s390 stack frame sheet.
2520IBM Mainframe Channel Attachment a technology brief from a CISCO webpage
2521Various bits of man & info pages of Linux.
2522Linux & GDB source.
2523Various info & man pages.
2524CMS Help on tracing commands.
2525Linux for s/390 Elf Application Binary Interface
2526Linux for z/Series Elf Application Binary Interface ( Both Highly Recommended )
2527z/Architecture Principles of Operation SA22-7832-00
2528Enterprise Systems Architecture/390 Reference Summary SA22-7209-01 & the
2529Enterprise Systems Architecture/390 Principles of Operation SA22-7201-05
2530
2531Special Thanks
2532==============
2533Special thanks to Neale Ferguson who maintains a much
2534prettier HTML version of this page at
2535http://penguinvm.princeton.edu/notes.html#Debug390
2536Bob Grainger Stefan Bader & others for reporting bugs