What a bastion is in the cloud, and how to use one well on AWS — ProxyJump measured, SSM Session Manager, EC2 Instance Connect Endpoint

A building with one door

Put servers in a private subnet and the internet cannot reach them. That is the point. But operators have to get in. The classic answer to this contradiction is the bastion host: one small host in a public subnet, the only door open to the internet is its SSH port, and every other server accepts connections only from the bastion.

In building terms, you reduce the entrances to one and post a guard at that one door. With one guard, the entry log lives in one place too. In exchange, if that door is breached the whole building is. So the bastion becomes the host you harden the most, patch the most often, and log the most carefully.

A classic bastion is one line of SSH

OpenSSH has this pattern built in. The ssh_config manual describes ProxyJump like this:

Setting this option will cause ssh(1) to connect to the target host by
first making an ssh(1) connection to the specified ProxyJump host and
then establishing a TCP forwarding to the ultimate target from there.

SSH to the jump host first, then open a TCP forward from there to the final target. On the command line it is -J. I tried it in the homelab: from my Mac (192.168.219.101) through cubi01 (192.168.219.120) into nuc1 (192.168.219.116).

ssh -J cubi01 nuc1 'hostname; echo $SSH_CONNECTION'

The first attempt failed.

channel 0: open failed: connect failed: Temporary failure in name resolution
stdio forwarding failed

With -v you can see what ssh does.

debug1: Setting implicit ProxyCommand from ProxyJump: ssh -v -W '[%h]:%p' cubi01
debug1: Executing proxy command: exec ssh -v -W '[nuc1]:22' cubi01
Authenticated to cubi01 ([192.168.219.120]:22) using "publickey".
debug1: channel_connect_stdio_fwd: nuc1:22

The name nuc1 is in my Mac's /etc/hosts but not on cubi01 (getent hosts nuc1 returns nothing there). The target's name is resolved by the jump host, not by your laptop. It is the first fact anyone using a bastion runs into. Switch to the IP, and add HostKeyAlias so the existing known_hosts entry by name is used, and it goes through.

ssh -o HostKeyAlias=nuc1 -J cubi01 192.168.219.116 'hostname; echo $SSH_CONNECTION'

nuc1
SSH_CONNECTION=192.168.219.120 53994 192.168.219.116 22

Compare with a direct connection, no jump:

SSH_CONNECTION=192.168.219.101 55330 192.168.219.116 22

The target server sees the connection as coming from the bastion (…120). My Mac's address (…101) appears nowhere. From the target's logs alone you only learn that "cubi01 logged in". Who actually logged in exists only in the bastion's logs. That one line is the reason a bastion's audit log must be kept and shipped elsewhere.

This homelab has five nodes inside 192.168.219.0/24, and the only things open from the internet are the gateway's ports 80 (for ACME HTTP-01) and 443. Operational commands run from inside the LAN via ssh cubi01. So cubi01 is an admin node on the LAN, not a bastion visible from the internet. The moment you have to open port 22 to the internet is when you have a real bastion, and from that moment every worry above becomes real.

What you take on by running your own bastion

• Inbound port 22. SSH open to the internet gets hammered all day. The open door is itself the attack surface.
• Key management. When someone leaves, authorized_keys on the bastion and on every target needs editing. In practice it does not happen.
• Patching. The bastion is an ordinary EC2 instance in a public subnet. Kernel and OpenSSH vulnerabilities must be patched there first.
• A logging blind spot. As shown above, targets see only the bastion's address. Without session recording on the bastion, "who did what" disappears.
• A single point of failure. If the bastion dies nobody gets in. Run two and you have two things to manage.

AWS writes this list into its documentation nearly verbatim and offers two alternatives.

AWS's answer 1: SSM Session Manager

The Systems Manager documentation introduces Session Manager as node management "without the need to open inbound ports, maintain bastion hosts, or manage SSH keys". The SSM Agent inside the instance connects outward to Systems Manager endpoints, and the operator rides that connection in. No inbound rule is needed at all.

aws ssm start-session --target i-0123456789abcdef0

Permissions are entirely IAM. Who may enter which instance is decided by tags or resource ARNs; session content goes to S3 or CloudWatch Logs, and API calls to CloudTrail. With no key files there are no departed employees' keys to remove.

Port forwarding works too. Pull the instance's port 80 to 56789 on your laptop,

aws ssm start-session --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["80"], "localPortNumber":["56789"]}'

or use the instance as a stepping stone to reach the RDS behind it. The remote host does not need to be registered with Systems Manager.

aws ssm start-session --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["mydb.example.us-east-2.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["3306"]}'

To keep using plain ssh and scp, put a ProxyCommand in ~/.ssh/config. Port 22 on the node stays closed (in the documentation's words, "You can close inbound ports on the node").

# SSH over Session Manager
Host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
    User ec2-user

One trap is stated explicitly in the documentation. Sessions established through SSH or port forwarding are not logged. SSH encrypts again inside the TLS tunnel, so Session Manager acts only as a tunnel. "Who connected to which instance and when" is in CloudTrail, but "what commands were typed" is recorded only for plain sessions (bare start-session). If auditing is the goal, SSH via Session Manager can be denied in IAM, and the documentation gives the policy.

{
  "Effect": "Deny",
  "Action": "ssm:StartSession",
  "Resource": "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
}

AWS's answer 2: EC2 Instance Connect Endpoint

Session Manager needs an agent inside the instance. For instances that cannot take the agent, or when you simply want ordinary SSH or RDP, AWS has the EC2 Instance Connect Endpoint (EICE). The documentation calls it an "identity-aware TCP proxy". Create an endpoint in one subnet of the VPC, and a tunnel authenticated with your IAM credentials opens from your computer to that endpoint, from which TCP goes out to instances inside the VPC.

Properties confirmed in the documentation:

• Instances need no public IP, and the VPC needs no internet gateway.
• Every connection attempt, successful or not, is logged to CloudTrail.
• There is no additional cost (only cross-AZ data transfer when connecting to an instance in another Availability Zone).
• Only one per VPC and per subnet, and each endpoint supports up to 20 concurrent connections.
• A tunnel lasts at most 1 hour, and the IAM condition maxTunnelDuration can force it shorter. Even if the IAM credentials expire first, the tunnel persists until the limit.
• It is for management traffic. High-volume transfers are throttled.
• By default the instance sees the endpoint ENI's address as the client. With client IP preservation on it sees the real address, but that is IPv4-only and same-VPC only.

Using it is just a different ProxyCommand.

Host i-*
    ProxyCommand aws ec2-instance-connect open-tunnel --instance-id %h

Or let the CLI do it.

aws ec2-instance-connect ssh --instance-id i-0123456789abcdef0 --connection-type eice

For Windows instances, open the tunnel with --remote-port 3389 and point an RDP client at the local port.

The three side by side

How to use it on AWS

The first choice is Session Manager. No inbound port, no keys, and commands are recorded. All five burdens the bastion carried disappear. Attach the SSM Agent and a role with AmazonSSMManagedInstanceCore to the instance. If the private subnet has no internet route, add VPC endpoints for Systems Manager (PrivateLink) and the agent connects inside the VPC.

When you need SSH itself, use EICE. Work that needs a real TCP connection — rsync, scp, remote development in an IDE, RDP — opens a tunnel through EICE and uses the usual tools. Remember the limits of one per subnet, 20 concurrent, one hour, and tie it shorter with maxTunnelDuration.

A bastion EC2 is the last choice. If you must run one, keep the two things the measurement taught. Targets see only the bastion's address, so ship the bastion's sshd log out (CloudWatch Logs, for example), and let the targets' security groups allow port 22 only from the bastion's security group. Put the SSM Agent on that bastion too, so that the day you close port 22 there is still a way in.

In one line

A bastion is the idea of "reduce the doors to one", and on AWS the best realisation of that idea is no longer a bastion host but Session Manager, entered through IAM with no inbound port. Use EICE when you need real SSH, and choose a self-run bastion last, without forgetting that targets see only the bastion's address.