Troubleshooting

---

Applies to SSHepherd 2.0

Where to find logs

Component	Log
Agent on Linux	sudo journalctl -u shepagent
Agent on Windows	C:\ProgramData\sshepherd\shepagent.log (not written at the default logging level), and the Windows Application event log
shepctl and the Control App	.shepctl in your home folder (~/.shepctl/ on Linux, %USERPROFILE%\.shepctl\ on Windows)
C3 server	microk8s kubectl -n sshepherd-<label> logs statefulset/python-flask

At their default levels, the Agent and the C3 server log very little. To see more:

• Agent: set Logging Level on the agent’s Settings tab to a more detailed level, reproduce the problem, then set it back.
• C3 server: raise Logging Level on Settings → General. Saving restarts the C3 server for about 2.5 minutes, so do this in a maintenance window, and restart the Agents afterwards.

Agents

An agent shows as connected, but tunnels fail with “Agent not connected”

This can happen after the C3 server restarts or is upgraded, or after Settings → General is saved. Restart the Agent service on the host:

• Linux: sudo systemctl restart shepagent
• Windows: Restart-Service shepagentsvc in PowerShell as administrator

Make this part of every C3 upgrade or restart: wait until the C3 server is fully up, then restart all Agents.

An agent shows as offline

1. Check the Agent service on the host (systemctl status shepagent, or Services on Windows).
2. Check that the host can reach the C3 server on TCP 443: curl -s -o /dev/null -w "%{http_code}\n" https://your.sshepherd.server/healthz/ready
3. Check the Agent log for certificate or authentication errors.

After a reboot, the Agent reconnects by itself, usually within a minute.

Registration fails on Windows with a DISM error

The Agent couldn’t install OpenSSH Server. Install it yourself, then register with --nowinssh. See Installing the Agent.

Registration after a reinstall fails with “PermissionError” and “last_peer”

The old configuration in C:\ProgramData\sshepherd is blocking registration.

A removed agent’s Windows service still shows “Running”

After you remove an agent from the C3 server, its Windows service can keep running without connecting. Uninstall the Agent, or register it again to add the host back with a new host ID.

Tunnels

shepctl says “tunnel open”, but connections close immediately

The agent is refusing the traffic. Check:

• For --relayport: the port is in the agent’s Allowed Ports.
• For a tunnel definition: its Remote Host and Remote Port are filled in. Editing a Generic definition can clear its Remote Port; see Tunnel Definitions.
• For an Agent-to-Agent tunnel: Remote Host is 127.0.0.1.

To find the reason, raise the agent’s Logging Level and check its log.

“Tunnel error: Agent is in full isolation mode”

The agent’s Availability is Fully Isolated, or Isolated with Admin Access and you don’t have the connect-admin-isolated-tunnel permission. See Agent Isolation.

“Tunnel error: Unauthorized” or “Tunnel not found”

• Unauthorized: the tunnel definition isn’t assigned to you or your group.
• Tunnel not found: the definition has been deleted. Definitions are also deleted when their group or agent is deleted.

A tunnel to a host the user can’t reach hangs

If you open a tunnel with --hostid to an agent you don’t have access to, shepctl can wait without an error. Check shepctl list-host: you can reach only the agents it lists.

RDP or Windows connections fail when many are opened at once

Windows agents accept only about five new connections per minute, and each RDP sign-in uses two. Space connections at least 15 seconds apart. See Creating an RDP Session.

An RDP session dropped

Start the tunnel again on the same local port, and the Remote Desktop client reconnects to the same Windows session. See Creating an RDP Session.

Scripts don’t detect failed commands

shepctl returns exit code 0 even when a command fails. Check the command’s output instead.

Certificates

Clients or agents report certificate errors

• Make sure the certificate matches the C3 server’s host name and includes its intermediate certificates. See Creating the C3 Server SSL Certificate.
• If your certificate comes from an internal certificate authority, add that authority to the trusted certificates on each client and agent.
• --ignore-ssl-errors skips the check, but use it only in test environments.

Administrator Console

Saving Settings → General fails

• An error that mentions volumeClaimTemplates means the server was upgraded without the new rbac.yaml. Apply it, then save again: microk8s kubectl -n sshepherd-<label> apply -f rbac.yaml
• If the form asks for a mail user name and password, fill in the mail settings, or clear the mail port. See Configuring C3 Settings.

Forgot Password or Reset User Password fails

Mail isn’t set up. Configure SendGrid or an SMTP server. See Configuring C3 Settings.

You can’t sign in after setting up SSO

Use the local sign-in page: https://your.sshepherd.server/#/signin?auth=standard. See Configuring a Third-Party Identity Provider.

A user can’t sign in after a role was deleted

Deleting a role that users still hold leaves those users unable to sign in or run commands. To avoid this, remove all users from a role before deleting it.

System Status shows Healthy, but something isn’t working

System Status shows only the database and API pods, and doesn’t refresh by itself. Check all the C3 server’s pods with:

microk8s kubectl -n sshepherd-<label> get pods
microk8s kubectl -n sshepherd-ingress-rtmp get pods

Recordings

RDP sessions aren’t recorded

1. Check that Enable Recording is on for the agent, and restart the Agent service after changing Windows recording settings.
2. Check that the Windows agent can reach the C3 server on TCP 1935.
3. On Windows Server 2016, fresh sign-ins aren’t recorded; reconnects are. See Session Recording.
4. If recordings stopped after a C3 upgrade, the recording service’s configuration may have been reset. Restore it from the server bundle: sudo ./nginx-rtmp-config.sh sshepherd-<label> <label>

An RDP session shows a black player

The agent wasn’t recording when the session took place. Sessions without a recording can show a player that never loads.

Installing the C3 server

Message	What to do
CPU lacks AVX	Use a CPU with AVX. On a virtual machine, pass the host CPU’s features through to the guest.
less than 16GB of RAM	Give the server more memory. A VM with exactly 16 GB reports 15 GB; use at least 17 GB.
MicroK8s v1.35 or later is required	Update MicroK8s: sudo snap refresh microk8s --channel=1.36/stable.
Kubernetes ValidatingAdmissionPolicy support is required	Use a Kubernetes version that supports ValidatingAdmissionPolicy.
Namespace ... already exists	A C3 server with that host name is already installed. Don’t delete the namespace if it holds data; contact FullArmor support.
Failed to authenticate with registry	Check the registry host, user and password FullArmor gave you.
Let's Encrypt certificate was not issued in time	Check that public DNS points to the server and that ports 80 and 443 are reachable from the internet.

Getting help

When you contact FullArmor support, include:

• The SSHepherd® versions of the C3 server, Agent (shepagent version) and shepctl (shepctl version).
• What you did, what you expected, and what happened, with exact error messages.
• The relevant logs, with passwords, keys and tokens removed.