IBM Support

How to Set Up a Hayes Compatible Modem

Question & Answer


How to Set Up a Hayes Compatible Modem


The following document provides an overview of how to set up an Asynchronous Modem. This is applicable to operating system level 4.x and 5L.

An overview of setting up an asynchronous modem
Physically attaching the modem
Create a tty on the chosen port
Adjusting the Devices file
Programming the modem
Testing the modem
Problem determination
Modem to modem communication failure
Appendix A
Appendix B
Appendix C
Appendix D

An overview of setting up an asynchronous modem

The following actions are required:

  1. Physically attach the modem.
  2. Create a tty device on the port.
  3. Adjust the devices file on the system.
  4. Program the modem.
  5. Test the modem.

Physically attaching the modem

Attach a direct (modem) cable between the modem and any open serial port on the system.

Common options

  • native serial port on back of system
  • 8 port adapter attachment
  • 16 port adapter attachment
  • 64 port adapter attachment
  • 128 port adapter RAN

NOTE: Although a modem can be attached to a 16 port adapter it is not recommended. The 16 port adapter had RTS looped back to CTS at the connector (physical RTS and CTS lines do not run between the system and the modem as they do in the other options). This renders RTS (hardware) flow control impossible resulting in possible data loss, communication errors, or no connection ability at all.

All options require a direct cable but each adapter has different pin-outs and requires a different connector on that direct cable (most modems are db-25 connected but there are some variations on the market).

NOTE: See Appendix C, "Modem cable pin outs" for various adapter attachments.

Create a tty on the chosen port

NOTE: See Appendix A, "Add/Change/Show a tty", for help with this function.

Special notes:

  1. Set flow control to RTS/CTS (hardware): not Xon/Xoff (software)flow control.
  2. If using 8 wire cable off of a RAN port, set Alternate RJ-45 to enable.
  3. Take note of the baud rate speed, i.e. 9600, it will be needed later.
  4. Take note of the tty that SMIT assigns the port, i.e. ttyl, it will be needed later.
  5. Verify that enable login is set to disable so a getty will not be started on the port.

Adjusting the Devices file

  1. Log in as root.
  2. Edit /etc/uucp/Devices file using vi or any other file editor.

    This file has many commented out lines. Append a new line to the bottom of the file.

    NOTE: There cannot be any spaces or characters before the word "Direct". Do not hit "Enter", add lines or spaces after "baud direct" when typing the command in the following step.

  3. Type the following command:
        Direct ttyx - baud direct

    (Where x is put the tty number assigned to the port by SMIT and replace "baud" with the speed of the tty you noted when SMIT created the tty. Such as Direct tty1 - 9600 direct)

  4. Save this file.

Programming the modem

Modems typically must be programmed differently from their factory defaults. The factory defaults are often tailored to the PC environment where you typically do not dial into your PC. Likewise in a PC environment the dial out process is typically driven by the user. There is no real standard for modem programming. The term "Hayes Compatible" may simply mean that the modem uses the "AT" command and supports the basic commands of the original Hayes modem. You should also not assume that the programming requirements are the same for all modems from the same manufacture. It's most common that differing models from a manufacture will have differing program requirements.

Most modern modems are Hayes compatible, so they will accept at commands to program their registers.

To program a modem, type the following on the command line:

    cu -ml ttyx (x is the number of the modem tty)(Response should be 
    at          (Response read "OK")
    at&f        (Response should read "OK")
    at&cl       (Response should read "OK")
    at&d2       (Response should read "OK")
    ats0=1      (Response should read "OK")
    atq2        (Response should read "OK")
    at&w        (Response should read "OK") ~[tilde].[period[ (represents 

After hitting enter on the last command, the system will close the connection to the modem and give you a command line prompt. You are ready to test.

NOTE: See Appendix D, "Modem strings" for some modem specific strings.

Testing the modem

  1. Test the modem for dial out operation, type the following at the command line:
        cu -ml ttyx  (x is your tty number)
        (Response should read "CONNECTED")
        at  (Response should read "OK")
  2. In the following command, replace the 0's with whatever phone number you use to dial into another system. Put no spaces or dashes(-)in the phone number. You may use a comma(,)for a pause: i.e. 9,11234567890. When you hit enter after the phone number, the cursor will go back under the a of atdt, and you will not get "OK" from your modem.

    In about 30 sec to 1 min something like the following will appear on your screen:

        CONNECTED AT 9600

    This will scroll up and you will see the login prompt from the called system on your screen. Login in as normal and carry out whatever operation you need to do.

  3. Test a modem for dial in operation, type the following at the command line:
        penable ttyx  (x is your tty number)

    This will start a getty (login prompt) running on the modem tty. Now call the phone number of the line attached to the modem on the system and you will get the login herald on your calling computer's screen.

  4. Finally, set the tty Login Enable to your desired setting (enable, disable, share, delay) permanently, via SMIT. For help with the "Add/Change/Show a TTY" function see Appendix A.

    The following chart lists the possible setting and their meaning.

/etc/getty (login herald) running on tty
no getty running on tty
/etc/getty associated with tty but not run on port until CD goes high (Normally a result of a modem dialup connection occurring)
same condition as share except the herald is not seen on the tty terminal until the "Enter" key is pressed

Problem determination

Cannot CU out to the modem

The following is a list of responses the system may give other than Connected when you enter cu -ml tty1 on the command line.

NOTE: We will use tty1 and a baud of 9600 for our examples in this text. Type the following from the command line:

    cu -ml tty1
  (Response should read "CONNECTED")

The message Connection Closed -m flag not valid may occur on OS levels below 3.2.5. The -m flag tells the system to ignore missing CD and open the port. Lower OS levels do not have the -m flag option so we will need to use the Dialers file to send the modem the set up string of at commands.

  1. Open /etc/uucp/Dialers in vi or any other file editor.

    Go to the bottom of the file and enter this line on a new line by itself:

        set^^=,-,^^^"" at&c1&d2q1s0=1&w ""
        (the "^" represent spaces)

    Save the file.

  2. Open /etc/uucp/Devices in vi or any other file editor.

    Rather than a Direct line entry for the tty, enter the following on its own line:

        ACU tty1 - 9600 set

    Comment out all other tty entries by putting a pound sign at the front of each line.

  3. Save the file.
  4. Run the following command:
        cu -d 123

    This will produce on-line debugging and you can see the steps of the cu command on the screen. The command will find the line ACU tty1 - 9600 set in the Devices file. Then it will go to the Dialers file and find the set chat script we entered. You will see on the screen a series of expect, gotit, send them lines. It then sends the at string to the modem and then hangs because the script was not programmed to dial the bogus phone number 123.

  5. In the cu session to get back to the command prompt, type the following command:
    The modem has now been programmed. Edit /etc/uucp/Devices file and restore to original condition.

The "cu -ml tty1" command returns "No Devices Available"

The cu command response should be No Devices Available. This means the /etc/uucp/Devices file either does not contain the line, Direct tty1 - 9600 direct or the line is there, but it has a typo in it. The line must be exactly as follows:

    Direct tty1 - 9600 direct
          ^    ^ ^    ^
    (the "^" represent spaces)

If the line appears correct, there may be some non-printable typos in it. Delete the line and then add it again as previously outlined by typing:

The "cu -ml tty1" command returns "Cannot Access Device" or "Device Locked"

The response will be Cannot Access Device or Device Locked. This means some other process has control of the tty. To find out what the process is, type from the command line:

    ps -ef | grep tty1

If the process is a getty, run pdisable tty1.

If it is determined that the process should not be running, kill the process.

NOTE: Some processes must not be stopped with the -9 option (i.e. slattach), take the appropriate steps to kill the process and free the port.

In the case of slattach for example you would first run:

    ifconfig sl(x) detach
    kill #  (# is process id of slattach)

If no process is returned except the grep itself, the port is hung and must be cleared.

To clear a hung port, see Appendix B, "Clearing a hung tty port".

The "cu -ml tty1" command returns "CU" not found

The response from the system should be cu not found This indicates a missing uucp file set.

Run the following command:

    lslpp -h *uucp*

If the system responds and the uucp fileset is not found on the system. Install the file set.

    For a 3.2 system install 'bosext1.uucp.obj'.
    For a 4.x system install ''.
    For a 5.x system install ''.

After the installation of the fileset, return to normal mode to set up procedure.

Modem to modem communication failure

Common dial in problems

  • The system gives a herald but asks for R\I\N\G's password. (or says you have logged in with all uppercase)

    This indicates the modem attached to the system has result codes turned on and is sending RING to the system when someone dials in.

    To fix the above:

    cu to the modem and send command atq2. If modem will not accept q2 send atq1. (After sending atq1 the modem will not respond with "OK" because you have turned all result codes off. The code atq0 will turn them back on). If the modem accepts atq2 but still asks for RINGs then send the password atq1.

  • If caller gets connected but does not see a herald run, type the following command:
    ps -ef | grep ttyx

If a getty is not running, type the following command:

    penable ttyx
then dial in again.

If you penable the getty and it dies or you see a console message of Init Respawning Too Rapidly, the carrier detect (CD) is high on the port. To verify this run the following command:

    /usr/lbin/tty/stty-cxma ttyx | pg

The output is a snapshot of the line conditions of the tty, (+) is high and (-) is a low. If you see CD+ and no one is dialed into the modem, this indicates a problem. Carrier detect should only go high when another modem calls the system. Either the modem is sending the CD or the system is altering the port itself. The quickest way to check where the problem lies is to turn the modem off, and run the following command again:

    /usr/lbin/tty/stty-cxma ttyx | pg

If you see CD- the modem is the problem. It must either be reprogrammed or it is not functioning properly and should be replaced. Try at&c1 or check the modem's manual for the proper command to have CD follow the line signal.

If however, you still see CD+ with the modem off, the system is the cause. Check the following:

  1. Unplug the cable to see if DTR is being looped back to CD on the port.

    NOTE: If you are using a 8 wire cable off a 128port adapter RAN unit there is an error in the original cable diagram, see Appendix C, "Modem cable pin outs" for correct pin to pin connections.

  2. Run:
        stty -a < /dev/ttyx | pg

    Look for the output clocal (not -clocal). If you see it listed go into SMIT and do a change/show on the tty. On the final screen you will see two lines called STTY runtime attributes and STTY login attributes. On both of the lines scan the list of attributes in the [] box to the left. Find and delete the clocal attribute found in either of the lines. Run the following command to make the change take effect:

        rmdev -l tty

    NOTE: If the system says the device is busy, run the following command:

        pdisable ttyx

    then repeat the following command:

        rmdev -l tty

    Type mkdev -l ttyx If you do not find clocal listed, check the tty settings in SMIT for either Force Carrier or Ignore Carrier being Enabled.

  3. If the tty is set to Share or Delay mode (for dial out and in operation) you run cu -ml tty1 and if it says cannot access device this indicates a getty is running on the port when it should not be there.

    Getty is only run on a SHARE/DELAY tty when someone dials into the modem (and CD goes HIGH) The problem is the same as in step 2 above. The port is seeing CD high when no one is dialed in. Troubleshoot the same way as step 2.

    NOTE: When the getty is running, it should not be associated with the tty until another modem dials in and CD goes high.

  • A common question on modems is "I have a 14.4 or a 28.8 speed modem but there is no option in SMIT for a tty speed to match this?" A 14.4 or 28.8 modem tty may be set to 19,200 or 38,400. Both types of modems can operate at faster speeds. Set the 'baud' in SMIT and in the Devices file.

    To set the 'baud' as stated in the modems manual or on the system, from the command line run:

        cu -s(baud) -ml ttyx

    For example if you run cu -s19200 -ml tty1 this will open the tty1 port at a speed of 19200.

    When you see Connected type on the command line:

    The response from the modem should read "OK".

    You have now communicated with the modem at the 19200 baud speed and the modem now matches the port speed.

    NOTE: Most modems run with v.32 compression. This can give up to a 4 to 1 compression. A 14,400 bits/secmodem compressing 4 bits to 1 bit results in 14,400 times 4 which is 57600 bits/sec. You can set a tty to run at 38400 and program the modem for the same speed because it could run faster than 38,400 bits/sec.

Common dial out problems

The most common dial out problem is getting connected and then getting garbage characters or nothing after the connect to the other modem.

The number one cause of this problem is a 'baud' rate mismatch between the modem and the tty it is attached to.

Three baud rates must match.

  • The rate set in SMIT for the tty.
  • The rate in the Devices file line.
  • The rate of the modem itself.

Look at the 'tty baud via smit', then run the following command:

    cu -s(baud) -ml ttyx   (i.e. cu -s9600 -ml tty1)

This will open the port at the baud rate of the -s flag on the command line. You should see the response Connected.

From the command line type:

    at  (You should get the response "OK")

You have now set the modem to the -s flag speed.

If you get an error on this command run, cu -ml tty1. When you get Connected, set the baud rate with the proper at command for your particular modem.

NOTE: If you cu to modem and get Connected but the cursor does not respond to key strokes or you do not get an "OK" then enter the following: (You will not be able to see what you typed):

    ate1 "enter" atq0 "enter";
These two commands should turn the echo and answer codes back on in the modem.

If you still do not get screen activity, it could be due to one of the following reasons:

  1. The modem is off or bad.
  2. The cable is the wrong type or bad.

    For example if you can see the TD light on the modem flicker when you are typing but you cannot see anything on your screen; this would indicate a bad modem or cable in the RD line because you can talk to the modem but it cannot send a signal back to the system.

  3. You have cu to the wrong tty.

    You may also have bad hardware in the setup (modem, cable, etc.). Check your hardware, especially a new setup that has never worked prior to this attempt.

Appendix A

Add/Change/Show a tty

To add a tty to the system, log on or su to root. Then type SMIT or smitty on the command line. Select the following order of options:


Remember to see the section on "Create a tty on the chosen port" for special settings that need to be made on the same screen where you enter the port number.

To "change/show" a tty you must be root user, then start SMIT and select the following options:


NOTE: On some 3.2 systems after you have selected the tty you wish to work with, there is a sub window giving you 4 options for different groups of attributes. The most commonly chosen are tty program and tty hardware characteristics. Select one of the four attributes for the final change/show to work with. If it is not in the group you chose, select one of the others.

Appendix B

Clearing a hung tty port

There are numerous reasons that cause a port to hang or to try and free them up again.

  1. From the command line type:
        fuser -k /dev/ttyx
    This sends a hangup signal to all processes registered on the port. Or try:
        /usr/lbin/tty/stty-cxma flush ttyx
  2. If you are running on a 3.2.5 system run:
        echo tty ttyx | crash | more
    This will give a screen output. Take note of two pieces of information:
    1. sid 0x0000ABCD (ABCD will be some hex number, convert the hex to decimal). It will be the process using the port, kill this process.
    2. If the sid is all 0's but the following statement is open

    This means the port is stuck open but there is no process to kill.

  3. Physically unplug the serial cable from the port on the system that goes to the device and repeat the following commands:
        fuser -k /dev/ttyx

    NOTE: The following command is done as a last resort before rebooting the system because if something is hung this also may hang while trying to work with the odm. If that happens it will require a reboot.

        /usr/lbin/tty/stty-cxma flush ttyx

    If none of these attempts result in freeing the tty, so a cu can be run, the only option is to re-boot the system. (This is especially true in the case of /usr/lbin/tty/stty-cxma flush ttyx, having no pid number but a reading of open on the port.)

  4. Type the following at the command line:
        rmdev -l ttyx
        mkdev -l ttyx
    This step will remove and re-add the tty which may clear the port.
  5. As a final step, attempt to perform a hardware flush. If the hung tty is on a 128port adapter, use one of the arrow keys on the RAN to display the physical port that the tty is on. Then simultaneously push both left and right arrow keys. This will do a hardware flush of that port.

Appendix C

Modem Cable Pin Outs

        Cable-B         Adapter-to-DB25 S1/S2 Serial Port
                        PN00G0943               FFC 252
        10 9 8 7 6                      SIGNAL     SYSTEM     DEVICE
    ------------------                  NAME        END        END
    |_   * * * * *  _|
      |  * * * * * |                    TXD          1          2
      --------------                    DTR          2          20
         5 4 3 2 1                      RTS          3          4
      10-pin rectangular                RI           4          22
      (system end) FEMALE               NOT USED     -          -
                                        RXD          6          3
                                        DSR          7          6
                                        CTS          8          5
    1                      13           DCD          9          
  __________________________            SIG GND     10          
 \. . . . . . . . . . . . . 
  \. . . . . . . . . . . . /
   14                    25
         (device end)
             CABLE-D         ASYNC EIA-232 MODEM CABLE
             PN6323741       10FT            FFC 259
   13                        1                     System    Device
                                        Signal     End       End
  \ O O O O O O O O O O O O O /       Shield Gnd   Shell     Shell,1
   \ O O O O O O O O O O O O /           TD          2---------2
    -------------------------            RD          3---------3
    25                     14            RTS         4---------4
     DB-25 FEMALE CONNECTOR              CTS         5---------5
          (system end)                   DSR         6---------6
                                      Sgnl Grnd      7---------7
     1                       13          CD          8---------8
    ___________________________          DTR        20---------20
   \ . . . . . . . . . . . . . /         RI         22---------22
    \ . . . . . . . . . . . . /
     14                     25
            (device end)
              CABLE - P RJ-45 TO DB25    CONVERTED CABLE
              PN59F3432                  FFC266
         64port CONCENTRATOR
           RJ-45 CONNECTOR
              |       |
______________|_______|___________       Signal
|                                |       Name      Concentr   Device
|                                |
| {}  {}  {}  {}  {}  {}  {}  {} |       Shld Gnd     1         1
----------------------------------       RTS          2         4
  1    2   3   4   5   6   7   8         RD           3         3
                                         CD           4         8
                                         Sgnl Gnd     5         7
     1                       13          TD           6         2
    ___________________________          DTR          7        20
   \ . . . . . . . . . . . . . /         CTS          8         5
    \ . . . . . . . . . . . . /
     14                     25
            (device end)
            (64port adapter to modem off concentrator)
            CABLE NK 10-PIN TO DB25 DIRECT RAN CABLE PN43G0935
   Remote Async Node End
        RJ45  10-pin
                                              Signal       Remote
                                              Name         Async   Device
                  _________                                RJ45    DB25
                  |       |
__________________|_______|________________   RI             1      22
|                                         |   DSR            2       6
|                                         |   RTS            3       4
| {}  {}  {}  {}  {}  {}  {}  {}  {}  {}  |   Chassis Gnd    4     Shell
 ------------------------------------------   TD             5       2
  1   2   3   4   5   6   7   8   9   10      RD             6       3
                                              Signal gnd     7       7
                                              CTS            8       5
                                              DTR            9      20
        1                       13            CD            10       8
      \ . . . . . . . . . . . . . /
       \ . . . . . . . . . . . . /
         14                     25
            DB-25 MALE CONNECTOR
                (device end)

This cable is not interchangeable with the 64-port RJ45 to DB25 Cable-P

                 10 PIN RAN DIRECT (MODEM) CABLE

Appendix D

Modem Strings

The following are modem strings that work for certain modems.

NOTE: In all cases following, if the modem gives an error on the whole string, enter each register separately as shown in "Program the modem" section.

Use ATL5  ATL6  and ATL7 to view the settings:
Change each register individually i.e.
AT$MB9600 or ATB1 or AT$SB9600
Set the registers as follows:
B1 E1 M1 Q0 R0 V1 X0 &E1 &E4 &E6 &E8 &E10 &E13 &E15
    $MB9600 $SB9600 $BA0 &W0
    S0  S2  S3  S4  S5  S6  S7  S8  S9  S10  S11  S13
    001 043 013 010 088 002 045 002 017 007  070  037
    S24  S25  S30
    020  001  000
    $A0 &A0 #A0 &B0 &BS1 &C1 $D0 &D2 #DB0 $EB0 %E1 #F0 $F1
    &G0 #L0 $MI0 &M0 &P0 #P2 &Q0 $R0 &R1 &RF1 &S0 
$SP0 &SF0
    #T1 $V0 $V6 $VD0 &X0 Y0 $MB9600 $SB9600 $BA0 &W0

NOTE: In both cases be sure to set the no. 3 and no. 8 dip switches down and the rest up.

The second string sets the modem to use Xon (software flow control). Use AT&F1 rather than AT&F2 to set to RTS (hardware flow control)

All the above cases were done on individual modems and IBM systems. None are guaranteed to work in any particular case. They are only guide lines to be used. Consult your modem manual for additional information or call the modem manufacture, for additional set up and support.

[{"Product":{"code":"SWG10","label":"AIX"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"Attached devices","Platform":[{"code":"PF002","label":"AIX"}],"Version":"5.3;5.2;5.1;4.3","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}},{"Product":{"code":"SWG10","label":"AIX"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"Network communications","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Historical Number


Document Information

Modified date:
17 June 2018