Installing the Command-and-Control (C3) Server

Summary

The Command-and-Control Console (C3) server is the heart of SSHepherd^®. Both Agents and Control Clients connect to the C3 server. And the C3 server, like an old telephone operator, facilitates connecting the Control Client’s socket to the Agent’s socket.

The SSHepherd^® C3 Server is a NGINX Ingress Controller with role-based access control (RBAC) and a Mongo DB. This server acts as a Bastion/Proxy hybrid to manage ingress/egress, routing, and session requests. It also stores the historical and live streaming session recordings and provides remote session monitoring and termination capabilities.​

The C3 is built to be deployed into a Kubernetes (k8s) cluster to enable rapid scaling/descaling that may be required. The C3 is comprised of the following parts inside the k8s system:

• WebUI/API container(s)
  • WebUI Host – hosts the Administrator Console
  • Socket Routing – actual routing of data between sockets
• RTMP Container(s)
  • This container is used to record, save and playback the RDP sessions
• Persistent Volume Store
  • Used to store recordings
• NGINX Ingress Container(s)
  • Handles the routing of traffic to the specific containers
• MongoDB database container(s)
  • Used to store data about users, agents, and recordings

IMPORTANT: It is possible to move the MongoDB database/cluster out of k8s and instead, use either a stand-alone or managed/hosted version of MongoDB. To do this, you won’t be able to use either of the quick deployment scripts (as they assume you want a new MongoDB replica set inside the k8s cluster).

Before You Start

The C3 server can be installed into just about any k8s deployment. Below we will cover 2 more specific examples, Microsoft Azure AKS and MicroK8s. However, the C3 server has also been successfully deployed to Amazon EKS, K3s by Rancher, and many others.

IMPORTANT: if you are deploying to k8s other than Microsoft Azure AKS or MicroK8s, it may require more knowledge of k8s than what this guide will cover.

Full Armor will provide:

• SSHepherd^® license
• Container Registry Credentials (for installing the C3 Server)
• Repository and Repository Credentials (for installing Linux shepctl and shepagent)
• Link to download:
  • Windows Control App/shepctl and shepagent install package
  • MacOS Control App/shepctl install package

If you have not received the items above, contact your sales rep for assistance.

You will also need the following:

• SSL Certificate. You will need PEM versions of your SSL certificate and SSL key. If you have a pfx and/or pkcs12 certificate file, see either the README-aks or README-microk8s file to see how to convert them to the proper format. Please see the Creating the SSL Certificate and Certificate Requirements article for help creating a certificate and/or to verify your certificate file meets the requirements for the C3 server installation.
• DNS entry for the host name you intend to use for your cluster (e.g., sshepherd.fullarmor.com)
• By default, automatic updates will be applied to your C3 server. If you would like to manage this manually, please see Upgrading SSHepherd^® C3 Server before you begin installation.
• If you intend to utilize the Bulk User Creation feature, you will need to enable user requested password resets (forgot password feature). There are two ways to enable this:

1. Using SendGrid
2. Using a standard SMTP Server

See the Enabling Password Reset Feature for more information.

Installing on MicroK8s

MicroK8s is a lightweight Kubernetes package that comes with installers for every major operating system. The easiest way to install it is via the snaps plugin on an Ubuntu system and/or any other Linux system that supports snaps (https://microk8s.io/docs/getting-started). It can also be installed on many other systems as well (https://microk8s.io/docs/install-alternatives). 

Some things you’ll need before you start:

1. A Linux based host (or WSL2 in Windows) for configuring your cluster via kubectl
2. Credentials to the SSHepherd^® container registry (contact your sales rep if you have not received these)

Included with the SSHepherd^® C3 .tar file (shepserver-install-<version>.tar) are some helper scripts and a quick readme. These will allow you to quickly deploy the C3 server with default options. Those included files are:

• README-microk8s
  • Simple readme file on how to use the shepserver-microk8s.sh file
• shepserver-microk8s.sh
  • Script that will install a standard install of all of the C3 components
  • Must be run on the machine that MicroK8s is installed on
• pvc-microk8s.yaml
  • Yaml file to define the needed persistent volume claim
• sshepherd.yaml
  • Yaml file that defines all of the k8s components that should be deployed
• microk8s-ingress.yaml
  • Yaml file that defines in microk8s how to configure the ingress
• microk8s-ingress-patch.yaml
  • Patch to the nginx ingress in sshepherd.yaml

Using the script and the Let’s Encrypt service to auto-generate your certificates

1. If firewall is enabled on your Microk8s system, open ports – 80*, 443, 1935 (TCP)

* Port 80 is required when using the Let’s Encrypt service to auto-generate your certificates

2. Copy the file to your system

• shepserver-install-<version>.tar

3. Untar the file

tar -xf shepserver-install-<version>.tar

4. Change to the shepserver-install-<version> directory

cd shepserver-install-<version>

5. Enable running of the script

sudo chmod 744 shepserver-microk8s.sh

If you already have microk8s installed, skip to Step 8.

6. Install microk8s

sudo snap install microk8s --classic

7. Join the Group

sudo usermod -aG microk8s ${USER}

8. Re-enter the session for the group update to take place

su - $USER

9. SSHepherd requires the following microk8s addons – dns, hostpath-storage, ingress. Run this command to ensure they are installed

microk8s enable hostpath-storage ingress dns

10. Change to the shepserver-install-<version> directory

cd shepserver-install-<version>

11. Now you are ready to run the script

$ ./shepserver-microk8s.sh <hostname> <registry host> <registry user> <registry password> letsencrypt <your email>

Example: ./shepserver-microk8s.sh docs.fasshep.us sshepdev.azurecr.io sshepdev FB10B8XrbuUU=85lk73zvG9T8en/Itt/+ letsencrypt shepadmin@widgetco.com

12. Update your DNS with the IP address of your microk8s system. Ensure that a DNS entry exists pointing <hostname> to the microk8s host. The Administrator Console will not load unless a hostname is used in the URL. https://<hostname>

13. Wait for DNS to propagate, then Verify connectivity

Using the script and your own SSL Certificate

1. If firewall is enabled on your Microk8s system, open ports – 443, 1935 (TCP)
2. Copy the files to your system
   • shepserver-install-<version>.tar
   • your .key and .crt files (Contact your sales rep if you need assistance)

If using PFX certificate, convert to .crt and .key:
openssl pkcs12 -in my-cert.pfx -clcerts -nokeys -out my-cert.crt
openssl pkcs12 -in my-cert.pfx -nocerts -out my-cert.key

Get unencrypted key:
openssl rsa -in my-cert.key -out my-cert-unencrypted.key

3. Untar the file

tar -xf shepserver-install-<version>.tar

4. Enable running of the script

sudo chmod 744 shepserver-microk8s.sh

If you already have microk8s installed, skip to Step 8.

5. Install microk8s

sudo snap install microk8s --classic

6. Join the Group

sudo usermod -aG microk8s ${USER}

7. Re-enter the session for the group update to take place

su - $USER

8. SSHepherd requires the following microk8s addons – dns, hostpath-storage, ingress. Run this command to ensure they are installed

microk8s enable hostpath-storage ingress dns

9. Now you are ready to run the script

$ ./shepserver-microk8s.sh <hostname> <registry host> <registry user> <registry password> cert <crt file> <unencrypted key file>

Example: ./shepserver-microk8s.sh docs.fasshep.us sshepdev.azurecr.io sshepdev FB10B8XrbuUU=85lk73zvG9T8en/Itt/+ cert docs-fasshep-us.crt docs-fasshep-us.key

10. Update your DNS with the IP address of your microk8s system. Ensure that a DNS entry exists pointing <hostname> to the microk8s host. The Administrator Console will not load unless a hostname is used in the URL. https://<hostname>

IMPORTANT: If your certificate does not match the hostname, or it’s not trusted by your endpoints, you can add –ignore-ssl-errors to the shepagent and shepctl commands. The –ignore-ssl-errors option should only be used in evaluation and test environments. Do not use this option in production.

11. Wait for DNS to propagate, then Verify connectivity

The following video will guide you through installing Microk8s on Ubuntu, followed by installing the SSHepherd^® Command-and-Control (C3) Server using the provided installer script, and then installing the SSHepherd^® Control (shepctl) to create your initial Administrator account.

Do-It-Yourself

1. Create a namespace in MicroK8s for SSHepherd^®

microk8s kubectl create namespace sshepherd

2. Configure Helm

microk8s helm3 repo add fa-bitnami https://charts.bitnami.com/bitnami
microk8s helm3 repo update

3. Install MongoDB

NOTE: If you are going to use an existing MongoDB replicaset or a managed/hosted one, skip to Step 5.

microk8s helm3 install mongodb fa-bitnami/mongodb -n sshepherd --set architecture=replicaset --set replicaCount=3

IMPORTANT: Step 3 could take several minutes to complete, as all 3 pods must be running before moving forward to Step 4. You can monitor the progress with the following command.

microk8s kubectl -n sshepherd rollout status statefulset/mongodb

4. Get your MongoDB credentials

mongoPassword=$(microk8s kubectl get secret --namespace sshepherd mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)

5. Create a secret the API service will use to connect to MongoDB

IMPORTANT: If you are using a different MongoDB than the one deployed above, change the connection string below with the correct one.

microk8s kubectl -n sshepherd create secret generic mongodb-url --from-literal=SECRET_MONGODB_URL="mongodb://root:$mongoPassword@mongodb-0.mongodb-headless.sshepherd.svc.cluster.local:27017,mongodb-1.mongodb-headless.sshepherd.svc.cluster.local:27017/sshepherd?authSource=admin&replicaSet=rs0"

6. Create your TLS secret for ingress

NOTE: Replace my.key and my.crt in the command below with the file names of your SSL certificate

microk8s kubectl -n sshepherd create secret tls aks-ingress-tls --key my.key --cert my.crt

7. Create secret to access container registry

NOTE: Replace ‘$registry’, ‘$username’ and ‘$password’ with the registry, username and password you were provided.

microk8s kubectl -n sshepherd create secret docker-registry sshepherd-acr --docker-server="$registry" --docker-username="$username" --docker-password="$password"

8. Create a couple more secrets for the cluster

There is a cluster secret and a password salt that needs to be generated for use inside of the k8s cluster. Running the following will create a random value for both.

clusterSecret=$(openssl rand -hex 64)
passwordSalt=""

for i in {0..40}; do
passwordSalt=$passwordSalt$(shuf -i 1-9 -n 1 | sed -r 's/\s+//g')
done

microk8s kubectl -n sshepherd create secret generic cluster-secret --from-literal=SECRET_CLUSTER_SECRET="$clusterSecret"
microk8s kubectl -n sshepherd create secret generic password-salt --from-literal=SECRET_PASSWORD_SALT="$passwordSalt"

9. Wait for MongoDB

Ensure that the MongoDB cluster is all the way up before continuing. There should be 4 pods for MongoDB and they must be running – mongodb-0, mongodb-1, mongodb-2, mongodb-arbiter-0. It could take 3-5 minutes for them to get all the way up.

IMPORTANT: You can skip this step if you are using MongoDB outside of the k8s cluster.

microk8s kubectl -n sshepherd get pods

10. Update sshepherd.yaml

sshepherd.yaml needs to be updated with the proper registry and cluster hostname. Replace ‘$registry’ with the registry you were provided. Replace ‘$hostname’ with the FQDN of the hostname you plan to use (ex. sshepherd.fullarmor.com)

sed -i "s/hostname-here/$hostname/g" sshepherd.yaml
sed -i "s/registry-here/$registry/g" sshepherd.yaml

11. Deploy the SSHepherd^® Server

microk8s kubectl -n sshepherd -f pvc-microk8s.yaml apply
microk8s kubectl -n sshepherd -f sshepherd.yaml apply
microk8s kubectl -n ingress -f microk8s-ingress.yaml apply
microk8s kubectl -n ingress patch ds nginx-ingress-microk8s-controller --patch "$(cat microk8s-ingress-patch.yaml)"

12. Wait for all of the pods to start

microk8s kubectl -n sshepherd get pods

13. Update your DNS with the IP address of your microk8s system. Ensure that a DNS entry exists pointing <hostname> to the microk8s host. The Administrator Console will not load unless a hostname is used in the URL. https://<hostname>
14. Wait for DNS to propagate, then Verify connectivity

Verify connectivity

1. After DNS propagation completes, connect to your host at https://hostname using a web browser.
2. You should be prompted to login to SSHepherd^®. This tells you that the SSHepherd^® C3 Server installation was successful. However, before you can login, you will create your initial Admin account.

Congratulations, your SSHepherd^® Command and Control (C3) Server is now up and running.

Create the Initial Admin Account

To login to the Administrator Console, we need to create the initial Administrator account. The first user account created in SSHepherd^® is automatically your main Admin account.

In order to do this, you will install the SSHepherd^® Control CLI (shepctl) and execute a simple command.
Below are the steps to do this:

Installing the SSHepherd^® Control CLI

Ubuntu/Debian

IMPORTANT: Replace user and password combinations in the following commands with your repository credentials.

sudo apt update && sudo apt install -y wget gnupg apt-transport-https

wget -O - https://user:password@repo.sshepherd.io/fullarmor.gpg.key|sudo apt-key add -

echo deb https://repo.sshepherd.io/debian buster main | sudo tee /etc/apt/sources.list.d/fullarmor.list

echo machine repo.sshepherd.io login user password password | sudo tee /etc/apt/auth.conf.d/fullarmor.conf

sudo apt update

sudo apt install -y shepctl

RedHat/CentOS

IMPORTANT: Replace user and password combinations in the following commands with your repository credentials.

sudo yum install wget

wget -O - https://user:password@repo.sshepherd.io/el$(rpm -E '%{?rhel}')/sshepherd.repo | sudo tee /etc/yum.repos.d/sshepherd.repo

echo "username=user" | sudo tee -a /etc/yum.repos.d/sshepherd.repo

echo "password=password" | sudo tee -a /etc/yum.repos.d/sshepherd.repo

sudo yum update

sudo yum install shepctl

1. Download and install shepctl.msi
2. Once the SSHepherd^® Control (shepctl) is installed, open a Command Prompt as Administrator and type the following command to connect to your SShepherd^® C3 Server and create the initial administrator account. When prompted, enter a username (in email format) and password to create the account.

IMPORTANT: Replace <yourSSHepherdServer> with the hostname of your SSHepherd^® C3 Server.

shepctl --url https://<yourSSHepherdServer> create-user

3. Now login using the credentials you just created.

shepctl --url https://<yourSSHepherdServer> login

4. Confirm you are logged in. The output should be your newly created admin account.

shepctl whoami

5. Go back to your web browser and login to the SSHepherd^® Admin Console using the Administrator account you just created.

MacOS

IMPORTANT: In order to run RDP Sessions using SSHepherd^® on MacOS, Microsoft Remote Desktop must be installed. It is available in the Apple App Store.

1. Download and double-click shepctl.dmg
2. Remove the quarantine attribute (required on some MacOS installs). Open a terminal and type:

sudo xattr -r -d com.apple.quarantine /Applications/SSHepherd\ Control.app

3. Drag the SSHepherd^® icon and drop it over the Applications folder in the pop-up window

4. Once the SSHepherd^® Control (shepctl) is installed, open Terminal and type the following command to connect to your SSHepherd^® C3 Server and create the initial administrator account. When prompted, enter a username (in email format) and password to create the account.

IMPORTANT: Replace <yourSSHepherdServer> with the hostname of your SSHepherd^® C3 Server.

shepctl --url https://<yourSSHepherdServer> create-user

5. Now login using the credentials you just created.

shepctl --url https://<yourSSHepherdServer> login

6. Confirm you are logged in. The output should be your newly created admin account.

shepctl whoami

7. Go back to your web browser and login to the SSHepherd^® Admin Console using the Administrator account you just created.