====== CheckMK ======
[[https://checkmk.com|CheckMK]] is a great monitoring solution that can be used for free if you want to know what is going on with e.g. your [[Puppet]] server or your own [[independence|Mail- and Cloud server]].
If you want to use it for free, there are two different options. Either you use CheckMK Raw, which does not offer all features of the Enterprise Edition, or you use CheckMK Free which offers all features but is limited to a certain number of hosts/services. I prefer Open Source Software whenever possible, so I use CheckMK Raw for my needs. For a comparison of the available editions you should check out the [[https://checkmk.com/pricing|official site]].
The [[https://docs.checkmk.com/latest/en/|documentation]] of CheckMK is superb and I really urge you to use it. Here I just want to describe some specialties which are not included in the standard CheckMK documentation or distribution.
===== CheckMK Agent over SSH =====
In older versions of CheckMK it was not possible to encrypt the traffic between the CheckMK server and the agents. Nowadays that is possible by registering agents to the server and factually running the agent over SSH is no longer necessary.
However, there might be situations when that is not feasible---e.g. when your network infrastructure allows only one-way traffic. That is why I try to document here how to still run the agent over SSH, obviously that has been removed in the official documentation.
==== On the CLI of the monitoring server ====
First of all, log in the the CLI of your CheckMK server and switch to the user of the monitoring site you created or want to run over SSH. There, you should create an SSH key like this:
ssh-keygen -t ed25519
While you are still logged in as your monitoring user, adjust the SSH configuration to accept host keys of new hosts only. This provides some level of security for [[https://en.wikipedia.org/wiki/Man-in-the-middle_attack|MITM attacks]], but also will break monitoring on hosts which change their SSH host key for whatever reason, please keep that in mind.
To do so, edit the ''~/.ssh/config'' file and add this line (or if it is already there, adjust it accordingly):
StrictHostKeyChecking accept-new
Have a look at the public SSH key you just created with ''cat ~/.ssh/id_ed25519.pub'' and copy it to the clipboard.
==== On all servers to be monitored ====
Now we are done on the CLI of the monitoring server and we need to configure all the servers which need to be monitored this way. Note that I do this using [[Puppet]] and you can use any means you wish (e.g. manually or using another state configuration tool than Puppet or use Ansible, the possibilities are endless.) but this description is about configuring one exemplary server manually.
On the server to be monitored, we first need to make sure that root can not login to the server, but at least run a restricted command over SSH. Also we do not want to allow passwords as a means of authentication. To accomplish this, at least these lines need to be configured in ''/etc/ssh/sshd_config'':
PermitRootLogin forced-commands-only
PasswordAuthentication no
I assume your regular authentication to log in to the CLI is by using SSH keys and not a password. Otherwise be careful with the ''PasswordAuthentication'' as you will no longer be able to log in by password after this change!
After adjusting the SSH server settings, don't forget to restart it using ''systemctl restart sshd''.
Now we take the public SSH key of the monitoring user we created earlier and compose a line like this:
command="/usr/bin/check_mk_agent" ssh-ed25519
That line needs to be added to root's ''~/.ssh/authorized_keys'' file. If that file does not exist (which it shouldn't), please create it like this (as root):
mkdir -m 0700 ~/.ssh
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
After creating the file enter the line we composed above.
Because we do not want the CheckMK agent to respond to network requests at all, you should stop and disable the ''cmk-agent-ctl-daemon.service'':
systemctl stop cmk-agent-ctl-daemon.service
systemctl disable cmk-agent-ctl-daemon.service
==== In the CheckMK GUI ====
At last, we are ready to tell CheckMK to run the Agent over SSH. To accomplish this, go to ''Setup -> Agents -> Other Integrations''.
Here, create a rule using the ''Add rule'' button. Fill in the values as follows.
{{ :apps:check-mk_agent.png?direct&400|}}
=== Under "Rule Properties" ===
* **Description** - A meaningful description, e.g. "CheckMK via SSH"
* **Comment** - A comment with further explanations, e.g. "CheckMK via SSH and key-based authentication"
=== Under "Individual program call instead of agent access" ===
* **Command line to execute** - Fill in ''ssh -T root@$HOSTADDRESS$'' here.
=== Under "Conditions" ===
* **Condition type** - "Explicit Conditions"
* **Folder** - "Main" or any folder that you want to apply this to.
You could also apply this rule to hosts with certain tags or labels, but in general this should be all. Save and activate the rule and do a service discovery on an arbitrary host to check if it works as designed.
===== Puppet =====
There used to be a great CheckMK plugin to monitor the Puppet Agent and its status on a system. Usually this is automatically done by the Puppet Enterprise Console, but if you are using the OpenSource version of Puppet you are out of luck and have to find a way to monitor that yourself.
On the [[https://exchange.checkmk.com/|CheckMK Exchange]] there are three Plugins available. Two seem to be dead, just one of the plugins seems to be still alive. And the one which is still alive is based on the Bakery---i.e. the commercial CheckMK version.
If I see this correctly, writing a plugin for CheckMK is not overly complicated so I might be inclined to write my own check plugin in future. As soon as I find the time for that I will of course update this website, so stay tuned!
~~DISCUSSION~~