Topic
  • 4 replies
  • Latest Post - ‏2012-05-25T05:19:34Z by Vilius
Vilius
Vilius
80 Posts

Pinned topic sshd autostart problem

‏2012-05-23T21:58:39Z |
Hello,

AIX 6.1
sshd worked fine.
Then I upgraded my AIX from 6100-02-04-0920 to 6100-03-10-1119 successfully(no errors, nothing unusual). After reboot sshd was not started, 3 errors in errpt:

# lssrc -s sshd Subsystem         Group            PID          Status sshd             ssh                           inoperative   # errpt | more A6DF45AA   0524002012 I O RMCdaemon      The daemon is started. 1BA7DF4E   0524002012 P S SRC            SOFTWARE PROGRAM ERROR CB4A951F   0524002012 I S SRC            SOFTWARE PROGRAM ERROR CB4A951F   0524002012 I S SRC            SOFTWARE PROGRAM ERROR 2BFA76F6   0524001812 T S SYSPROC        SYSTEM SHUTDOWN BY USER

Extended error view:

--------------------------------------------------------------------------- LABEL:          SRC_TRYX IDENTIFIER:     1BA7DF4E   Date/Time:       Thu May 24 00:20:49 GMT+03:00 2012 Sequence Number: 2237 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            PERM WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions DETERMINE WHY SUBSYSTEM CANNOT RESTART   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9020 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'369' FAILING MODULE sshd --------------------------------------------------------------------------- LABEL:          SRC_RSTRT IDENTIFIER:     CB4A951F   Date/Time:       Thu May 24 00:20:49 GMT+03:00 2012 Sequence Number: 2236 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            INFO WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9035 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'234' FAILING MODULE sshd --------------------------------------------------------------------------- LABEL:          SRC_RSTRT IDENTIFIER:     CB4A951F   Date/Time:       Thu May 24 00:20:47 GMT+03:00 2012 Sequence Number: 2235 Machine Id:      xxxxxxxxxxxx Node Id:         xxxx Class:           S Type:            INFO WPAR:            Global Resource Name:   SRC   Description SOFTWARE PROGRAM ERROR   Probable Causes APPLICATION PROGRAM   Failure Causes SOFTWARE PROGRAM   Recommended Actions VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY   Detail Data SYMPTOM CODE 65280 SOFTWARE ERROR CODE -9035 ERROR CODE 0 DETECTING MODULE 
'srchevn.c'@line:
'234' FAILING MODULE sshd ---------------------------------------------------------------------------

So I tried to start sshd manually:

# startsrc -s sshd 0513-059 The sshd Subsystem has been started. Subsystem PID is xxxxxx.

So if I start sshd manually after reboot - it allways starts and works fine. But not automatically. So I thought that my ssh was not ok(was a bit old). Reinstalled newest version:

# lslpp -L|grep ssh openssh.base.client     5.8.0.6102    C     F    Open Secure Shell Commands openssh.base.server     5.8.0.6102    C     F    Open Secure Shell Server openssh.license         5.8.0.6102    C     F    Open Secure Shell License openssh.man.en_US       5.8.0.6102    C     F    Open Secure Shell openssh.msg.en_US       5.8.0.6102    C     F    Open Secure Shell Messages - # # lslpp -L|grep ssl openssl.base            0.9.8.1801    C     F    Open Secure Socket Layer openssl.license         0.9.8.1801    C     F    Open Secure Socket License openssl.man.en_US       0.9.8.1801    C     F    Open Secure Socket Layer

That didn't help - still unable to autostart with 3 errors. All config options I tried look the same as on my other machines which autostart sshd just fine.

Any ideas ?

thanks,
Vilius
Updated on 2012-05-25T05:19:34Z at 2012-05-25T05:19:34Z by Vilius
  • Vilius
    Vilius
    80 Posts

    Re: sshd autostart problem

    ‏2012-05-24T07:18:14Z  
    This problem is somehow related to /dev/console.
    My console device is serial tty:
    
    # lscons /dev/tty0 # lsdev -c tty tty0 Available 01-S1-00-00 Asynchronous Terminal
    

    So if I connect to server's serial port using some kind of serial client(putty for example) - it's not even necessary to login, and reboot server while serial session is enabled - no errors sshd starts fine! But if I unplug serial cable (no serial session) and reboot - 3 errors and sshd no go.
    If I change /dev/console to something like /dev/lft0 - no problems sshd starts fine.

    Any ideas why this is happening ?
    I still hope to keep /dev/console redirected to serial tty for troubleshooting.

    thanks,
    Vilius
  • MatthewBourne
    MatthewBourne
    27 Posts

    Re: sshd autostart problem

    ‏2012-05-24T08:26:53Z  
    • Vilius
    • ‏2012-05-24T07:18:14Z
    This problem is somehow related to /dev/console.
    My console device is serial tty:
    <pre class="jive-pre"> # lscons /dev/tty0 # lsdev -c tty tty0 Available 01-S1-00-00 Asynchronous Terminal </pre>
    So if I connect to server's serial port using some kind of serial client(putty for example) - it's not even necessary to login, and reboot server while serial session is enabled - no errors sshd starts fine! But if I unplug serial cable (no serial session) and reboot - 3 errors and sshd no go.
    If I change /dev/console to something like /dev/lft0 - no problems sshd starts fine.

    Any ideas why this is happening ?
    I still hope to keep /dev/console redirected to serial tty for troubleshooting.

    thanks,
    Vilius
    Hello Vilius

    Is your sshd already configured to log to syslog? If you enable "debug" level logging to syslog within your sshd config file (/etc/ssh/sshd_config ?) then you might be able to see more clearly what is going wrong.

    M.
  • SystemAdmin
    SystemAdmin
    6902 Posts

    Re: sshd autostart problem

    ‏2012-05-24T17:08:33Z  
    Hello Vilius

    Is your sshd already configured to log to syslog? If you enable "debug" level logging to syslog within your sshd config file (/etc/ssh/sshd_config ?) then you might be able to see more clearly what is going wrong.

    M.
    Hello,
    Can you verify that the sshd daemon is not started by inetd instead of startsrc
    Grep sshd /etc/inetd.conf
  • Vilius
    Vilius
    80 Posts

    Re: sshd autostart problem

    ‏2012-05-25T05:19:34Z  
    Hello,
    Can you verify that the sshd daemon is not started by inetd instead of startsrc
    Grep sshd /etc/inetd.conf
    Hi,

    I believe my sshd is started from inittab line (no sshd in inetd):
    
    l2:2:wait:/etc/rc.d/rc 2
    

    About sshd syslog logging:
    
    # cat /etc/ssh/sshd_config ........ # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO LogLevel DEBUG ........   # cat /etc/syslog.conf ........ auth.debug /home/vilius/ssh1.log
    

    I'm not sure if my logging config is correct - however I was not able to troubleshoot sshd this way:
    After doing configurations above I restarted sshd and syslogd - logging begins - sshd start, stop, connects are logged, seems OK. But when errors happen during boot sshd just logs nothing and looks like logging stops till next error free boot..

    Anyways like I said problem happens only when /dev/console is /dev/tty - this is old POWER4 and it's serial devices are real "saX" devices not like "vsaX" on newer ones.

    V