Configuring Vault SSH Secret Engine

This guide demonstrates how to configure HashiCorp Vault's SSH Secret Engine to sign SSH certificates using ED25519 keys instead of RSA, specifically for z/OS Unix System Services (USS) environments.

Prerequisites

Note: This guide is compatible with HashiCorp Vault version 1.19.12.

The SSH Secret Engine allows Vault to act as a Certificate Authority (CA) for SSH certificates.

Vault CLI configured.
HashiCorp Vault installed and running (can be on distributed platform).
Appropriate permissions to enable secret engines and configure roles.
z/OS USS with OpenSSH installed and configured.
Network connectivity between Vault server and z/OS USS.

Before you begin

Ensure you have the following ready before starting:

On your local machine:

Vault CLI installed and configured.
Valid Vault authentication token with appropriate permissions.
SSH client installed (OpenSSH recommended).
Your SSH public key generated (preferably ED25519: ssh-keygen -t ed25519).

On Vault Server:

Vault server running and unsealed.
Network access to Vault API (typically port 8200).
Permissions to enable secret engines and create roles.

On z/OS USS Target Server:

SSH access with administrative privileges.
OpenSSH 7.4 or later installed (for ED25519 support).
z/OS V2R3 or later (for ED25519 support).
Write access to /etc/ssh/ directory.
Ability to restart SSH daemon.

Information You'll Need:

Vault server address (e.g., http://vault.example.com:8200).
z/OS USS hostname or IP address.
Your z/OS USS username.
Your Vault token or authentication credentials.

Procedure

Vault Server Steps (Steps 1-7): Perform these on a machine with Vault CLI access.
z/OS USS steps (See "z/OS USS Target Server Configuration" section): Perform these on the z/OS mainframe.
Client steps (Steps 8-9): Perform these on your local machine.

Configure Vault server:
1. Verify that Vault is running and accessible:
```
vault status
```
2. Check currently enabled secret engines:
```
vault secrets list
```
3. Enable the SSH secret engine at the path ssh-client-signer:
```
vault secrets enable -path=ssh-client-signer ssh
```
4. Generate a signing key for the CA using ED25519 algorithm:
```
vault write ssh-client-signer/config/ca \
  generate_signing_key=true \
  key_type=ed25519
```
  Key parameters:
  - generatesigningkey=true: Instructs Vault to generate a new signing key.
  - key_type=ed25519: Specifies ED25519 algorithm (more secure and faster than RSA).
5. Read back the CA configuration to verify the public key:
```
vault read ssh-client-signer/config/ca
```
  This will display the public key that needs to be added to target SSH servers.
6. Configure a role that defines certificate parameters:
```
vault write ssh-client-signer/roles/sshrole \
  allow_user_certificates=true \
  allowed_users="<username>" \
  key_type="ca" \
  default_user="<username>" \
  ttl="10m0s"
```
  Role parameters:
  - allowusercertificates=true: Enables user certificate signing.
  - allowed_users="<username>": Comma-separated list of allowed usernames.
  - key_type="ca": Specifies this is a CA-based role.
  - default_user="<username>": Default username if not specified in signing request.
  - ttl="10m0s": Certificate validity period (10 minutes).
7. Sign a user's public key using the Vault API:
```
curl \
  --header "X-Vault-Token:<your-vault-token>" \
  --request POST \
  --data '{
    "public_key": "<user'\''s public key>",
    "valid_principals": "<username>"
  }' \
  http://<vault-server-ip>:8200/v1/ssh-client-signer/sign/sshrole
```
  Request parameters:
  - public key: The user's SSH public key (typically from ~/.ssh/id ed25519.pub).
  - valid_principals: Username(s) the certificate is valid for.
  Response: The API returns a JSON response containing the signed certificate in the signed_key field.
8. Save the signed certificate from the API response to a file (for example, signed-key):
```
# Extract signed_key from JSON response and save to file
echo "<signed_key_from_response>" > signed-key
chmod 600 signed-key
```
9. Connect to the target server using the signed certificate:
```
ssh -o CertificateFile=signed-key <username>@<target-server>
```
  SSH option:-o CertificateFile=signed-key: Specifies the path to the signed certificate.

Configure z/OS USS target server to trust certificates signed by your Vault CA.

From a system with Vault CLI access, retrieve the CA public key from Vault:


vault read -field=public_key ssh-client-signer/config/ca > trusted-user-ca-keys.pem

Transfer the CA public key to z/OS USS using one of these methods:

Option A: Using FTP (Binary mode):


ftp <zos-hostname>
binary
put trusted-user-ca-keys.pem
quit

Option B: Using SFTP:


sftp <username>@<zos-hostname>
put trusted-user-ca-keys.pem
quit

Option C: Using SCP:


scp trusted-user-ca-keys.pem <username>@<zos-hostname>:/etc/ssh/

Log into z/OS USS and configure the SSH daemon:


# Navigate to SSH configuration directory
cd /etc/ssh

# Move the CA public key to the correct location (if not already there)
mv ~/trusted-user-ca-keys.pem /etc/ssh/trusted-user-ca-keys.pem

# Set proper permissions (critical on z/OS USS)
chmod 644 /etc/ssh/trusted-user-ca-keys.pem
chown SSHD:SYS1 /etc/ssh/trusted-user-ca-keys.pem

# Edit sshd_config
vi /etc/ssh/sshd_config

Add the following line to /etc/ssh/sshd_config:


TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Ensure the CA public key file is in ASCII format (not EBCDIC):


# Check file encoding
file /etc/ssh/trusted-user-ca-keys.pem

# If needed, convert from EBCDIC to ASCII
iconv -f IBM-1047 -t ISO8859-1 /etc/ssh/trusted-user-ca-keys.pem > /tmp/ca-keys-ascii.pem
mv /tmp/ca-keys-ascii.pem /etc/ssh/trusted-user-ca-keys.pem

Restart the SSH daemon to apply changes:


# Stop SSH daemon
/etc/ssh/sshd -k

# Start SSH daemon
/etc/ssh/sshd

# Or use the z/OS operator command
# From TSO/ISPF or console:
# F SSHD,SHUTDOWN
# S SSHD

Alternative using USS commands:


# If SSH is managed as a started task
ps -ef | grep sshd
kill -HUP <sshd-pid>

Test the SSH configuration:


# Check sshd configuration syntax
/etc/ssh/sshd -t

# Verify the CA key is readable
cat /etc/ssh/trusted-user-ca-keys.pem

# Check file permissions
ls -la /etc/ssh/trusted-user-ca-keys.pem

Automation example

For automated certificate signing, use the Vault CLI instead of Curl:


#!/bin/bash

# Sign the user's public key
vault write -field=signed_key \
  ssh-client-signer/sign/sshrole \
  public_key=@$HOME/.ssh/id_ed25519.pub \
  valid_principals="<username>" > signed-key

# Set proper permissions
chmod 600 signed-key

# Connect to server
ssh -o CertificateFile=signed-key <username>@<target-server>