Using AWS Instance Connect and Session Manager for simpler SSH access with Kamal

Even though Kamal is advertised as a tool to get you out of the big cloud, sometimes it’s just not that simple. Usually you have quite a bit of data in your cloud platform and you might not be ready to migrate all of that. Also, depending on the amount of services you’re using in your cloud provider you’ll need to audit what you’ll still need and what you can ditch.

You can still stay in the cloud though and use a more straightforward tool like Kamal on regular Ubuntu machines. Ditch the Beanstalk, ECS, EKS, blah, blah and just use simple EC2 instances. If you later on decide that you want to migrate to a simpler cloud like Digital Ocean or even co-locate your servers you’re already a few steps closer to doing that.

If you’re looking to deploy to AWS EC2 instances with Kamal there are some great options for not having to deal with SSH key management but still keep your servers secure. There are two main products for this and combining them both lets you keep access control at the IAM policy level which makes things very flexible.

The first product is EC2 Instance Connect which comes pre-installed on some of the newer and more common AMI images(Ubuntu 20.04, AL2023, etc.). Since it’s pre-installed it’s just a matter of configuring the correct IAM permissions for your users and then creating an instance connect endpoint in your VPC. You can either create the instance connect endpoint via the web UI or Terraform is another great option. You can only have one instance connect endpoint per VPC and they take about 5 minutes to provision them just as an FYI.

Part of the Instance Connect setup is to also add a security group rule to allow port 22 access just from within your network to your EC2 machine. Other than that all inbound access can be blocked off if you don’t need it, just ensure that your EC2 machine can make outbound requests. Your EC2 instance still needs to be able to make outbound connections so that it can check in with SSM and communicate with Instance Connect.

Here are the relevant bits for a Terraform plan to add the Instance Connect endpoint and allow access over port 22 from the connect endpoint into your EC2 instance. Modify your outbound rules as needed of course, this assumes you already have a basic VPC configured.

resource "aws_security_group" "allow_ssh" {
  name        = "Allow SSH"
  description = "allow ssh inbound traffic"
  vpc_id      = aws_vpc.main.id

  lifecycle {
    create_before_destroy = true
  }
}

# Allow ingress(inbound) access from the EIC security group
resource "aws_vpc_security_group_ingress_rule" "ssh" {
  security_group_id = aws_security_group.allow_ssh.id

  from_port   = 22
  to_port     = 22
  ip_protocol = "tcp"

  referenced_security_group_id = aws_security_group.eic.id
}

resource "aws_security_group" "eic" {
  name   = "EIC"
  vpc_id = aws_vpc.main.id

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_ec2_instance_connect_endpoint" "example" {
  subnet_id = aws_subnet.private[0].id
  security_group_ids = [
    aws_security_group.eic.id
  ]
}

# Allow all outbound traffic
resource "aws_security_group" "egress" {
  description = "allow outbound traffic"

  vpc_id = aws_vpc.main.id

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_vpc_security_group_egress_rule" "main" {
  security_group_id = aws_security_group.egress.id

  ip_protocol = "-1"
  cidr_ipv4   = "0.0.0.0/0"
}

# Create the web instance
resource "aws_instance" "web" {
  # Ubuntu 22.04 LTS HVM
  ami = "ami-008fe2fc65df48dac"

  instance_type = "t2.micro"
  associate_public_ip_address = true
  subnet_id                   = aws_subnet.public[0].id
  iam_instance_profile        = "SSMInstanceRole"
  vpc_security_group_ids = [
    aws_security_group.egress.id,
    aws_security_group.allow_ssh.id
  ]
}

The second AWS product is Session Manager and more specifically you’ll be using the Fleet Manager. I’ll let the AWS docs be the guide on getting session manager installed since it’ll be pretty specific to your organization but there’s also a default quick setup set of instructions as well. Session manager is a pretty powerful systems manager for AWS and various resources within your AWS account, not just EC2 instances. Session manager provides access to various resources without the need to open inbound ports, manage a bastion host, or SSH keys which really simplifies getting into an EC2 instance.

Once you have Session Manager up and running the last step is adding the IAM role you created when setting up Session Manager to the EC2 instance that you want to be able to connect to. If you did the quick setup it’ll be named something like SSMQuickSetup. Once the EC2 instance has this role it allows SSM to manage the instance.

You can modify the role for the instance by going to Actions -> Security -> Modify IAM role or by setting iam_instance_profile in the Terraform aws_instance resource for the machine.

It usually takes a few minutes up to about 30 minutes for the EC2 instance to check-in to Systems Manager after adding the IAM role. It’s usually easier to watch for the machine to come online in the web UI via Fleet Manager but you can also use the AWS CLI for this with aws ssm describe-instance-information. You won’t be able to connect via SSM until SSM can see the instance so you’ll need to wait until it shows up.

Once the EC2 instance is showing up in Fleet Manager and your EC2 Instance Connect endpoint is online you should be able to login to your machine without needing a key pair. We’ll push a key onto the server with instance connect but you don’t need a pre-installed key pair on the machine to be able to do this.

The connection into your EC2 instance works via an SSH ProxyCommand. We can first add it into ~/.ssh/config to test and then we should be able to move that command over to Kamal.

The ProxyCommand does two things.

1. Pushes a public key from your machine to your EC2 instance as an authorized key via the aws ec2-instance-connect send-ssh-public-key command. The key is only on the machine for 60 seconds and authorized for the user that you specify with the CLI command.

aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user ubuntu --ssh-public-key file://$HOME/.ssh/id_rsa.pub > /dev/null

1. With the short-lived key in place on the EC2 instance we can then utilize SSM to start a new SSH-based session. The SSH session will utilize our main ~/.ssh/id_rsa.pub key that we just pushed up when connecting via SSM so ensure that it’s been added to your local SSH agent(ssh-add ~/.ssh/id_rsa.pub) but it’s usually already there since it’s the default key.

aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=22'

We can then combine these two commands so that we push a short-lived key up to the server and then SSH via SSM to our EC2 instance. This also allows us to provision a keyless EC2 instance since we’ll just push up a key whenever we need access. We can also keep port 22 closed to the outside world which is great for security.

The full ProxyCommand then looks like this in your ~/.ssh/config. Once this is in place you can test the connection by SSH’ing via ssh ubuntu@instance-id-for-ec2-instance.

Host i-*
    ProxyCommand sh -c "aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user ubuntu --ssh-public-key file://$HOME/.ssh/id_rsa.pub > /dev/null & aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=22'"

Which we can then copy over to Kamal and now we don’t have to deal with SSH keys, just IAM permissions for AWS, yay!

service: hey
image: nickhammond/demo
servers:
  - i-27b5c3b
ssh:
  user: ubuntu
  proxy_command: aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user ubuntu --ssh-public-key file://~/.ssh/id_rsa.pub > /dev/null & aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=22'

Now we can test out the connection with Kamal which will use our newly defined proxy_command.

$ kamal lock status             
  INFO [46162f13] Running /usr/bin/env mkdir -p .kamal on i-27b5c3b
  INFO [46162f13] Finished in 3.033 seconds with exit status 0 (successful).
There is no deploy lock

There we go. No need to pass around a key amongst your team or add keys from your team to your EC2 instances. More importantly, no need for a bastion server. From here you can start locking down your IAM permissions to Instance Connect and Session manager on your users to whatever works best for your organization.