Topic
  • 21 replies
  • Latest Post - ‏2008-10-09T17:48:40Z by SystemAdmin
SystemAdmin
SystemAdmin
10114 Posts

Pinned topic gdb debugging problem

‏2007-07-01T14:57:53Z |
Hello everybody,
I have a couple of questions about gdb debugging on the simulator.
I am running the cell simulator (sdk 2.1) on an x86 system, Nad I followed the instructions written on the web page:
http://www.ibm.com/developerworks/library/pa-celldebug/

I used the first script in that page to install the 64-bit power pc version of gdb; I recompiled the 'simple' example i the tutorial of sdk with CFLAGS = -g; configured bogusne (the host system and the simulator can ping each other).
now, I launch on the simulator gdbserver (with the command: ppu-gdbserver :1000 ./simple) and launch on the host system the gdb program (with the command: powerpc64-linux-gdbtui simple).

The gdbserver correctly starts listening, giving me the output:
Process ./simple created; pid = 430
Listening on port 1000

on the host system, the gdb whows me the source code and its command propmpt.
but when I type, in the gdb console, the command: target remote 172.20.0.2:1000
I obtain the output:
REmote debugging using 172.20.0.2:1000
Couldn't establish connection to remote target
Roemte registry badly formatted: T0501:00000000fd00fbd0;40:00000000f7fe6660;
here: d00fbd0;40:00000000f7fe6660;

I tried to google te solution, but I really can't figure out where can be the problem... any help??

The second question is: as explained in the page previously linked, I'd like to "debug cell BE applications": I'd like to start the execution of a ppu program, and then attach gdb to the sou program when created by the ppu program.
The page suggests to set an enviroment variable, but once again, if I type:
  1. SPU_DEBUG_START=1 ./my_program &

The execution of the program doesn't stops at spu thread creation, as I'd expect. The program executes without any stop, reaching the end of the execution.
Again, where can be the problem? Am I missing to do something??

Thanks in advance for your help.
Francesco
Updated on 2008-10-09T17:48:40Z at 2008-10-09T17:48:40Z by SystemAdmin
  • mkistler
    mkistler
    551 Posts

    Re: gdb debugging problem

    ‏2007-07-01T20:41:05Z  

    Francesco,

    The procedures for debugging using gdbserver have changed a bit since that article was written. If you are using SDK 2.1, then you need to use the tools provided with that package (in particular, the ppu-gdb on the host system) when debugging in this fashion. The Programmer's Guide, specifically Chapter 3 Debugging Cell/B.E. applications, describes the procedures to follow for debugging apps on Cell SDK 2.1. You can find the programmer's guide in the pdfs directory of your SDK install, in a file named cpbprg00.pdf.

    Mike Kistler
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-07-02T09:40:35Z  
    • mkistler
    • ‏2007-07-01T20:41:05Z

    Francesco,

    The procedures for debugging using gdbserver have changed a bit since that article was written. If you are using SDK 2.1, then you need to use the tools provided with that package (in particular, the ppu-gdb on the host system) when debugging in this fashion. The Programmer's Guide, specifically Chapter 3 Debugging Cell/B.E. applications, describes the procedures to follow for debugging apps on Cell SDK 2.1. You can find the programmer's guide in the pdfs directory of your SDK install, in a file named cpbprg00.pdf.

    Mike Kistler
    Mike, thanks for your reply.
    I checked the pdf you suggested, and followed the instructions at the paragraph "setting up remote debugging"
    the steps are quite similar to the ones explained in the other document, except for the executable to launch on the client system, ppu-gdb.

    But the problem, persists. when I type, on ppu-gdb, the command "target remote 172.20.0.2:1000", I still obtain the output "remote registry badly formatted".

    on the simulator, the gdbserver prints out:
    Remote debugging from 172.20.0.1
    readchar: got EOF
    Remote side has terminated connection. GDBserver will reopen the connection.
    Listening on port 1000

    again, any help?
    thanks
    Francesco
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2007-07-02T16:43:26Z  
    Mike, thanks for your reply.
    I checked the pdf you suggested, and followed the instructions at the paragraph "setting up remote debugging"
    the steps are quite similar to the ones explained in the other document, except for the executable to launch on the client system, ppu-gdb.

    But the problem, persists. when I type, on ppu-gdb, the command "target remote 172.20.0.2:1000", I still obtain the output "remote registry badly formatted".

    on the simulator, the gdbserver prints out:
    Remote debugging from 172.20.0.1
    readchar: got EOF
    Remote side has terminated connection. GDBserver will reopen the connection.
    Listening on port 1000

    again, any help?
    thanks
    Francesco
    From the host, be sure you are using /opt/cell/bin/ppu-gdb and not /usr/bin/gdb.

    --
    IBM SDK Service Administrator
  • mkistler
    mkistler
    551 Posts

    Re: gdb debugging problem

    ‏2007-07-02T22:37:02Z  
    Mike, thanks for your reply.
    I checked the pdf you suggested, and followed the instructions at the paragraph "setting up remote debugging"
    the steps are quite similar to the ones explained in the other document, except for the executable to launch on the client system, ppu-gdb.

    But the problem, persists. when I type, on ppu-gdb, the command "target remote 172.20.0.2:1000", I still obtain the output "remote registry badly formatted".

    on the simulator, the gdbserver prints out:
    Remote debugging from 172.20.0.1
    readchar: got EOF
    Remote side has terminated connection. GDBserver will reopen the connection.
    Listening on port 1000

    again, any help?
    thanks
    Francesco
    Francesco,

    Well ... I've never seen this before. It looks to me like you may be connecting to the wrong system. I would check your IP configuration and route tables to make sure that the connection isn't going elsewhere. As an extreme, you could disconnect your system from the network totally, so only the host and sim are able to communicate. If none of these reveal the problem, I'm not sure I have any more advice to offer.

    Good luck!

    Mike Kistler
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-07-03T08:32:57Z  
    • mkistler
    • ‏2007-07-02T22:37:02Z
    Francesco,

    Well ... I've never seen this before. It looks to me like you may be connecting to the wrong system. I would check your IP configuration and route tables to make sure that the connection isn't going elsewhere. As an extreme, you could disconnect your system from the network totally, so only the host and sim are able to communicate. If none of these reveal the problem, I'm not sure I have any more advice to offer.

    Good luck!

    Mike Kistler
    I solved it.

    No, the problem wasn't on the network. The two systems can communicate (at network level), the problem was in the format of the registry. Looking for the error string in google, I found out that the problem was about the format of binaries.

    On the host side, seems that gdb expects 32-bits register, something
    like:

    T0501:ffffe6b0;40:40010470;

    but the remote gdbserver sends back 64-bits value:

    T0501:00000000ffffe6b0;40:0000000040010470;

    So it reports "Remote register badly formatted" error.

    In effect, the "simple" example was compiled with the -m32 flag.
    I tried to debug my own code, which is compiled with -m64 flag (but I edited the standard makefile, that should be quite similar to the one of the examples... don't know), and with that 64-bit code, everything works fine.

    Thanks for your time, anyway.
    Francesco
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-09-10T08:38:18Z  
    I solved it.

    No, the problem wasn't on the network. The two systems can communicate (at network level), the problem was in the format of the registry. Looking for the error string in google, I found out that the problem was about the format of binaries.

    On the host side, seems that gdb expects 32-bits register, something
    like:

    T0501:ffffe6b0;40:40010470;

    but the remote gdbserver sends back 64-bits value:

    T0501:00000000ffffe6b0;40:0000000040010470;

    So it reports "Remote register badly formatted" error.

    In effect, the "simple" example was compiled with the -m32 flag.
    I tried to debug my own code, which is compiled with -m64 flag (but I edited the standard makefile, that should be quite similar to the one of the examples... don't know), and with that 64-bit code, everything works fine.

    Thanks for your time, anyway.
    Francesco
    Beware of the distinction between
    ppu-gdbserver and
    ppu32-gdbserver on the system simulator.

    The latter is the one you need when compiling for 32 bit PPU. (which seems to be the default).
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-09-11T17:02:50Z  
    Beware of the distinction between
    ppu-gdbserver and
    ppu32-gdbserver on the system simulator.

    The latter is the one you need when compiling for 32 bit PPU. (which seems to be the default).
    Anyone understand the reason why SPE_START_DEBUG does not wait for spu-gdb to be launched and prompt it to continue with the created thread as the original poster found out (and as I found out trying to understand my issues with the dmabench application) ?

    That env. variable does do something, it just does not seem o put the SPE thread in the sleep state though...
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2007-09-11T18:54:00Z  
    Anyone understand the reason why SPE_START_DEBUG does not wait for spu-gdb to be launched and prompt it to continue with the created thread as the original poster found out (and as I found out trying to understand my issues with the dmabench application) ?

    That env. variable does do something, it just does not seem o put the SPE thread in the sleep state though...
    When using ppu-gdb in SDK 2.1, there is no need for spu-gdb (or SPE_START_DEBUG) anymore. From ppu-gdb, you can run "set spu stop-on-load" to break as each SPE program is started.

    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-09-11T21:59:55Z  
    • CellServ
    • ‏2007-09-11T18:54:00Z
    When using ppu-gdb in SDK 2.1, there is no need for spu-gdb (or SPE_START_DEBUG) anymore. From ppu-gdb, you can run "set spu stop-on-load" to break as each SPE program is started.

    --
    IBM SDK Service Administrator
    Is there no need (to use SPE_START_DEBUG) as in "it has no effect" or as in "it does not fully work as you'd expect it?".

    How else would it explain this behavior if it were just not doing anything:

    http://www.ibm.com/developerworks/forums/dw_thread.jsp?forum=739&thread=173629&cat=46

    ?

    So, what exactly does that env. variable do to a program ?
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-09-11T22:00:50Z  
    Is there no need (to use SPE_START_DEBUG) as in "it has no effect" or as in "it does not fully work as you'd expect it?".

    How else would it explain this behavior if it were just not doing anything:

    http://www.ibm.com/developerworks/forums/dw_thread.jsp?forum=739&thread=173629&cat=46

    ?

    So, what exactly does that env. variable do to a program ?
    Sorry, I apologize if the last post seemed a bit aggressive, it was not meant to be that way...
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2007-09-12T17:40:32Z  
    Sorry, I apologize if the last post seemed a bit aggressive, it was not meant to be that way...
    From your information it does appear to have an effect, it's just not something I've used since the combined debugger was released. That was the only use I saw for it before. I'll try to find out exactly what it does and get back to you.

    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2007-09-12T21:58:06Z  
    • CellServ
    • ‏2007-09-12T17:40:32Z
    From your information it does appear to have an effect, it's just not something I've used since the combined debugger was released. That was the only use I saw for it before. I'll try to find out exactly what it does and get back to you.

    --
    IBM SDK Service Administrator
    Thank you very much.
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2007-09-13T21:29:34Z  
    Thank you very much.
    Information I found:

    SPU_DEBUG_START and SPU_INFO were used to set the newly created SPU threads in a sleep state after creation. libspe2 then displayed a message how
    to connect to the SPU thread with spu-gdb via stderr. Today's combined debugger has different ways to attach to newly created SPU threads.
    For example GDB has a new command "set spu stop-on-load off" With this set to on the newly created SPU thread is also stopped by GDB itself.

    The code for SPU_DEBUG_START and SPU_INFO is still in libspe2 to support the libspe1 to 2 wrapper. So if you set these variables it should show the original behavior of stopping then SPU threads. But this is something that has not been tested (or supported) since the combined debugger was released.

    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2008-07-08T06:43:40Z  
    • CellServ
    • ‏2007-09-13T21:29:34Z
    Information I found:

    SPU_DEBUG_START and SPU_INFO were used to set the newly created SPU threads in a sleep state after creation. libspe2 then displayed a message how
    to connect to the SPU thread with spu-gdb via stderr. Today's combined debugger has different ways to attach to newly created SPU threads.
    For example GDB has a new command "set spu stop-on-load off" With this set to on the newly created SPU thread is also stopped by GDB itself.

    The code for SPU_DEBUG_START and SPU_INFO is still in libspe2 to support the libspe1 to 2 wrapper. So if you set these variables it should show the original behavior of stopping then SPU threads. But this is something that has not been tested (or supported) since the combined debugger was released.

    --
    IBM SDK Service Administrator
    Hi All,

    I have also this "register badly formatted" error message while trying remote debugging, and unfortunately cannot workaround it.
    My config is a PS3 with YDL6.0 and CellSDK3.0. On the PC side, I use Fedora7 with CellSDK3.0.
    It seems to me that it does not matter which gdbserver I use, both the 32 and 64 bit versions will send the register info in 64 bit format. The binary I am trying to debug has been compiled with -m32. The gdb client recognizes this and uses the appropriate format, but the server does not.
    Are there any (other) ways to remotely debug a 32 bit ppu program? Has anybody solved this? (Besides compiling the program with -m64 ;)

    By the way, what's the most important difference between 32 and 64 bit versions of the same program?
    How can I decide which one I need?

    Thanks for any help/hints.

    Regards,
    Balazs
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2008-07-08T15:38:40Z  
    Hi All,

    I have also this "register badly formatted" error message while trying remote debugging, and unfortunately cannot workaround it.
    My config is a PS3 with YDL6.0 and CellSDK3.0. On the PC side, I use Fedora7 with CellSDK3.0.
    It seems to me that it does not matter which gdbserver I use, both the 32 and 64 bit versions will send the register info in 64 bit format. The binary I am trying to debug has been compiled with -m32. The gdb client recognizes this and uses the appropriate format, but the server does not.
    Are there any (other) ways to remotely debug a 32 bit ppu program? Has anybody solved this? (Besides compiling the program with -m64 ;)

    By the way, what's the most important difference between 32 and 64 bit versions of the same program?
    How can I decide which one I need?

    Thanks for any help/hints.

    Regards,
    Balazs
    Are you using the ppu versions of each (and not just plain gdb )?

    For a 32 bit binary, you should be using ppu-gdb and ppu32-gdbserver.

    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2008-07-08T18:58:28Z  
    • CellServ
    • ‏2008-07-08T15:38:40Z
    Are you using the ppu versions of each (and not just plain gdb )?

    For a 32 bit binary, you should be using ppu-gdb and ppu32-gdbserver.

    --
    IBM SDK Service Administrator
    Yes, exactly.

    Compilation:
    
    
    root@ps3linux ~# /usr/bin/ppu32-gcc -o ppu_test ppu_test.o -L/usr/lib -L/opt/cell/sdk/usr/lib -m32 -Wl,-m,elf32ppc -R/opt/cell/sdk/usr/lib ../spu_embed/spu_embed.a -lspe2 -lpthread -lm


    Server side:
    
    
    root@ps3linux ~# ppu32-gdbserver --version
    GNU gdbserver 6.6.50.20070623-cvs
    ...
    This gdbserver was configured as "powerpc64-linux" root@ps3linux ~# ppu32-gdbserver localhost:10001 ./ppu_test


    Client side:
    
    
    user@pc-fc7 ~$ /opt/cell/toolchain/bin/ppu-gdb
    GNU gdb 6.6.50.20070627-cvs
    ...
    This GDB was configured as "--host=i686-pc-linux-gnu --target=powerpc64-linux".
    (gdb) target remote ps3linux:10001
    Remote debugging using ps3linux:10001
    Remote register badly formatted: T0501:00000000ff85aaa0;40:000000000ffd6684;
    here: f85aaa0;40:000000000ffd6684;
    (gdb)


    Balazs
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2008-07-08T19:20:38Z  
    Yes, exactly.

    Compilation:
    <pre class="jive-pre"> root@ps3linux ~# /usr/bin/ppu32-gcc -o ppu_test ppu_test.o -L/usr/lib -L/opt/cell/sdk/usr/lib -m32 -Wl,-m,elf32ppc -R/opt/cell/sdk/usr/lib ../spu_embed/spu_embed.a -lspe2 -lpthread -lm
    </pre>

    Server side:
    <pre class="jive-pre"> root@ps3linux ~# ppu32-gdbserver --version
    GNU gdbserver 6.6.50.20070623-cvs
    ...
    This gdbserver was configured as "powerpc64-linux" root@ps3linux ~# ppu32-gdbserver localhost:10001 ./ppu_test
    </pre>

    Client side:
    <pre class="jive-pre"> user@pc-fc7 ~$ /opt/cell/toolchain/bin/ppu-gdb
    GNU gdb 6.6.50.20070627-cvs
    ...
    This GDB was configured as "--host=i686-pc-linux-gnu --target=powerpc64-linux".
    (gdb) target remote ps3linux:10001
    Remote debugging using ps3linux:10001
    Remote register badly formatted: T0501:00000000ff85aaa0;40:000000000ffd6684;
    here: f85aaa0;40:000000000ffd6684;
    (gdb)
    </pre>

    Balazs
    The only way I can get this error is by using the wrong gdbserver.

    I'm assuming on the client you're actually calling ppu-gdb ppu_test and not just ppu-gdb, correct?

    Are you sure you're using the same binary on both machines?

    What does file ppu_test and ls -l ppu_test on each machine return?

    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2008-07-09T10:44:38Z  
    • CellServ
    • ‏2008-07-08T19:20:38Z
    The only way I can get this error is by using the wrong gdbserver.

    I'm assuming on the client you're actually calling ppu-gdb ppu_test and not just ppu-gdb, correct?

    Are you sure you're using the same binary on both machines?

    What does file ppu_test and ls -l ppu_test on each machine return?

    --
    IBM SDK Service Administrator
    Now I copied the compiled binary to the target and started the client with ppu_gdb ppu_test, but the result is the same.
    The file format is "ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped". The architecture on the client side is powerpc:common. (I don't know how to check this on the server side.)

    Meanwhile, I noticed that ppu32-gdbserver is an ELF 64-bit MSB executable, is this correct?
    (Currently I am using 64-bit versions of my programs and that works fine.)

    Balazs
  • CellServ
    CellServ
    1346 Posts

    Re: gdb debugging problem

    ‏2008-07-09T22:17:27Z  
    Now I copied the compiled binary to the target and started the client with ppu_gdb ppu_test, but the result is the same.
    The file format is "ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped". The architecture on the client side is powerpc:common. (I don't know how to check this on the server side.)

    Meanwhile, I noticed that ppu32-gdbserver is an ELF 64-bit MSB executable, is this correct?
    (Currently I am using 64-bit versions of my programs and that works fine.)

    Balazs
    That is not what I see, ppu32-gdbserver is 32-bit:

    1. file /usr/bin/ppu*-gdbserver
    /usr/bin/ppu32-gdbserver: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
    /usr/bin/ppu-gdbserver: ELF 64-bit MSB executable, cisco 7500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
    --
    IBM SDK Service Administrator
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2008-07-11T08:02:50Z  
    • CellServ
    • ‏2008-07-09T22:17:27Z
    That is not what I see, ppu32-gdbserver is 32-bit:

    1. file /usr/bin/ppu*-gdbserver
    /usr/bin/ppu32-gdbserver: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
    /usr/bin/ppu-gdbserver: ELF 64-bit MSB executable, cisco 7500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
    --
    IBM SDK Service Administrator
    I have reinstalled all the toolchain packages, now ppu32-gdbserver is 32-bit, and remote debugging works! :)

    Thank you for your patience.

    Balazs
  • SystemAdmin
    SystemAdmin
    10114 Posts

    Re: gdb debugging problem

    ‏2008-10-09T17:48:40Z  
    I have reinstalled all the toolchain packages, now ppu32-gdbserver is 32-bit, and remote debugging works! :)

    Thank you for your patience.

    Balazs
    I had the same problem, was in package ppu-gdb.ppc64.

    The ppu32-gdbserver provided with this package does not work properly with ppu-gdb on the host computer when debugging 32 bit applications. Removing ppu-gdb.ppc64 and installing only ppu-gdb.ppc solved the problem.

    (PS3 + YDL 6 + Cell SDK 3.0 manually installed)