Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
4 replies Latest Post - ‏2012-05-25T05:19:34Z by Vilius
Vilius
Vilius
78 Posts
ACCEPTED ANSWER

Pinned topic sshd autostart problem

‏2012-05-23T21:58:39Z |
Hello,

AIX 6.1
sshd worked fine.
Then I upgraded my AIX from 6100-02-04-0920 to 6100-03-10-1119 successfully(no errors, nothing unusual). After reboot sshd was not started, 3 errors in errpt:

# lssrc -s sshd Subsystem         Group            PID          Status sshd             ssh                           inoperative   # errpt | more A6DF45AA   0524002012 I O RMCdaemon      The daemon is started. 1BA7DF4E   0524002012 P S SRC            SOFTWARE PROGRAM ERROR CB4A951F   0524002012 I S SRC            SOFTWARE PROGRAM ERROR CB4A951F   0524002012 I S SRC            SOFTWARE PROGRAM ERROR 2BFA76F6   0524001812 T S SYSPROC        SYSTEM SHUTDOWN BY USER

Extended error view:

--------------------------------------------------------------------------- LABEL:          SRC_TRYX IDENTIFIER:     1BA7DF4E   Date/Time:       Thu May 24 00:20:49 GMT+03:00 2012 Sequence Number: 2237 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            PERM WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions DETERMINE WHY SUBSYSTEM CANNOT RESTART   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9020 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'369' FAILING MODULE sshd --------------------------------------------------------------------------- LABEL:          SRC_RSTRT IDENTIFIER:     CB4A951F   Date/Time:       Thu May 24 00:20:49 GMT+03:00 2012 Sequence Number: 2236 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            INFO WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9035 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'234' FAILING MODULE sshd --------------------------------------------------------------------------- LABEL:          SRC_RSTRT IDENTIFIER:     CB4A951F   Date/Time:       Thu May 24 00:20:47 GMT+03:00 2012 Sequence Number: 2235 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            INFO WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9035 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'234' FAILING MODULE sshd ---------------------------------------------------------------------------

So I tried to start sshd manually:

# startsrc -s sshd 0513-059 The sshd Subsystem has been started. Subsystem PID is xxxxxx.

So if I start sshd manually after reboot - it allways starts and works fine. But not automatically. So I thought that my ssh was not ok(was a bit old). Reinstalled newest version:

# lslpp -L|grep ssh openssh.base.client     5.8.0.6102    C     F    Open Secure Shell Commands openssh.base.server     5.8.0.6102    C     F    Open Secure Shell Server openssh.license         5.8.0.6102    C     F    Open Secure Shell License openssh.man.en_US       5.8.0.6102    C     F    Open Secure Shell openssh.msg.en_US       5.8.0.6102    C     F    Open Secure Shell Messages - # # lslpp -L|grep ssl openssl.base            0.9.8.1801    C     F    Open Secure Socket Layer openssl.license         0.9.8.1801    C     F    Open Secure Socket License openssl.man.en_US       0.9.8.1801    C     F    Open Secure Socket Layer

That didn't help - still unable to autostart with 3 errors. All config options I tried look the same as on my other machines which autostart sshd just fine.

Any ideas ?

thanks,
Vilius
Updated on 2012-05-25T05:19:34Z at 2012-05-25T05:19:34Z by Vilius
  • Vilius
    Vilius
    78 Posts
    ACCEPTED ANSWER

    Re: sshd autostart problem

    ‏2012-05-24T07:18:14Z  in response to Vilius
    This problem is somehow related to /dev/console.
    My console device is serial tty:
    
    # lscons /dev/tty0 # lsdev -c tty tty0 Available 01-S1-00-00 Asynchronous Terminal
    

    So if I connect to server's serial port using some kind of serial client(putty for example) - it's not even necessary to login, and reboot server while serial session is enabled - no errors sshd starts fine! But if I unplug serial cable (no serial session) and reboot - 3 errors and sshd no go.
    If I change /dev/console to something like /dev/lft0 - no problems sshd starts fine.

    Any ideas why this is happening ?
    I still hope to keep /dev/console redirected to serial tty for troubleshooting.

    thanks,
    Vilius
    • MatthewBourne
      MatthewBourne
      25 Posts
      ACCEPTED ANSWER

      Re: sshd autostart problem

      ‏2012-05-24T08:26:53Z  in response to Vilius
      Hello Vilius

      Is your sshd already configured to log to syslog? If you enable "debug" level logging to syslog within your sshd config file (/etc/ssh/sshd_config ?) then you might be able to see more clearly what is going wrong.

      M.
      • SystemAdmin
        SystemAdmin
        6902 Posts
        ACCEPTED ANSWER

        Re: sshd autostart problem

        ‏2012-05-24T17:08:33Z  in response to MatthewBourne
        Hello,
        Can you verify that the sshd daemon is not started by inetd instead of startsrc
        Grep sshd /etc/inetd.conf
        • Vilius
          Vilius
          78 Posts
          ACCEPTED ANSWER

          Re: sshd autostart problem

          ‏2012-05-25T05:19:34Z  in response to SystemAdmin
          Hi,

          I believe my sshd is started from inittab line (no sshd in inetd):
          
          l2:2:wait:/etc/rc.d/rc 2
          

          About sshd syslog logging:
          
          # cat /etc/ssh/sshd_config ........ # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO LogLevel DEBUG ........   # cat /etc/syslog.conf ........ auth.debug /home/vilius/ssh1.log
          

          I'm not sure if my logging config is correct - however I was not able to troubleshoot sshd this way:
          After doing configurations above I restarted sshd and syslogd - logging begins - sshd start, stop, connects are logged, seems OK. But when errors happen during boot sshd just logs nothing and looks like logging stops till next error free boot..

          Anyways like I said problem happens only when /dev/console is /dev/tty - this is old POWER4 and it's serial devices are real "saX" devices not like "vsaX" on newer ones.

          V