Documentation

• 1: License
• 2: Overview
• 3: Getting Started
• 3.1: Registration Portal
• 3.2: Easy Install
• 3.3: Advanced Install
• 4: Concepts
• 4.1: Image Scanning
• 4.1.1: Automating Scanning
• 4.1.2: Scanning in CICD Pipelines
• 4.1.3: Enforcing Compliance
• 4.1.3.1: Setting Up Policies
• 4.1.3.2: Installing Webhook
• 4.1.3.3: Enabling Enforcement
• 4.1.3.4: Exceptions
• 4.2: kube-bench
• 4.3: kube-hunter
• 4.4: KubeSec
• 4.5: Gatekeeper
• 4.5.1: Getting Started
• 4.5.2: Managing Constraints
• 4.5.3: Reviewing Exceptions
• 4.5.4: Reviewing Pod Compliance
• 4.6: Falco
• 5: Reference
• 5.1: Dash Parameters
• 5.2: Trawler Parameters
• 5.3: API Docs
• 6: Contributing to m9sweeper

1 - License

m9sweeper is licensed under the Apache License 2.0.

It was originally produced by Intelletive Consulting, owned by Jacob Beasley and Jason Woodman. Once feature complete, we chose to open source instead of selling it as a licensed product because we saw a huge need for the product and were deeply passionate about open source.

While the software is open source, but we do offer professional services for enterprises. This is ultimately how we fund ongoing support and maintenance for the product. Please contact us if you want information about a professional support contract. A support contract can include professional installation, support, monitoring, managed upgrades, and even coordinating security fixes and remediation in your environment. We also are available, at times, to do Kubernetes DevOps work.

We have released several open source products, including:

• m9sweeper
• m9sweeper helm charts
• kube-hunter easy install charts
• kube-bench easy install charts
• k8-ez - secure one-file kubernetes deployments for your apps

2 - Overview

What is it?

m9sweeper is an app for securing your kubernetes cluster. It does so by orchestrating a number of kubernetes security capabilities:

• CVE Scanning with Trivy
• Compliance with Gatekeeper
• CIS Benchmarking with Kube Bench
• Pen Testing with Kube Hunter
• Secure Deployments with KubeSec
• Intrusion Detection with Project Falco (Coming Soon)

If you are a beginner or experienced administrator, m9sweeper is a one-stop-shop to secure a kubernetes environment in a matter of hours, not days or weeks, as well as to continue to assess and report on your kubernetes security posture.

Why do I want it?

If you deploy apps on kubernetes, you should absolutely be making sure you are doing so securely.

m9sweeper makes it extremely easy to deploy and scan your cluster and its applications. It is the cheapest and fastest way to understand your security posture, and it walks you through securing a cluster step-by-step.

Where should I go next?

We recommend you proceed to the getting started guide.

3 - Getting Started

M9sweeper deploys as an app in your Kubernetes cluster. We like to say that is Kubernetes-native; that is to say, you can deploy it entirely inside of kubernetes. You do not need to install complicated applications in your Kubernetes nodes. This makes it very safe and easy to install.

It also can be installed in a hub-and-spoke way, with a single m9sweeper install monitoring multiple kubernetes clusters. This requires that you ingress traffic to m9sweeper, and is preferred if you have a lot of kubernetes clusters.

There are 3 ways to get started:

• Sign up for our registration portal, which will use a step-by-step wizard to guide you through the setup process
• Easy Install Guide - If you do not want a step-by-step wizard but still want to get it booted up with largely the defaults, start here.
• Advanced Install Guide - This covers all the options short of reviewing the reference materials.

3.1 - Registration Portal

To make installing m9sweeper as easy as possible, we have created a registration portal that lets you sign up your company to manage m9sweeper.

It includes a wizard that walks you through installing m9sweeper end-to-end, letting you just fill in the blanks.

When you are done, it generates a helm install command for you, which could be used to install it using our helm chart with a single command.

Register | Login

3.2 - Easy Install

You can install m9sweeper using our helm chart. This is a one-line cli command that installs m9sweeper. Change the default username/password to your own username/password and the API Key to something random/unpredictable.

helm repo add m9sweeper https://helm.m9sweeper.io/chartrepo/m9sweeper && \
helm repo update && \
helm upgrade m9sweeper m9sweeper/m9sweeper --install --wait --create-namespace --namespace m9sweeper-system \
  --set-string dash.init.superAdminEmail="super.admin@m9sweeper.io" \
  --set-string dash.init.superAdminPassword="password" \
  --set-string global.jwtSecret="changeme" \
  --set-string global.apiKey="YOUR-API-KEY"

Many more options are available. For serious enterprise deployments, we recommend creating a helm values.yaml file and versioning this in a code repository to make upgrades easier.

For more information, see the advanced install guide.

3.3 - Advanced Install

Install

License

To claim a free license go to m9sweeper licensing. Claiming a license is optional, but if you setup a license our online portal will enable remote monitoring. We can notify you, for example, if you a new version is available or if your environment is no longer being monitored for some reason.

Installation

We recommend putting your configuration in a values.yaml file and then deploying our app using helm. This example uses “helm upgrade –install”, which is an idempotent way of installing and/or upgrading the app. This is repeatable and the same command can be run regardless of whether you intend to upgrade or install the app.

helm repo add m9sweeper https://helm.m9sweeper.io/chartrepo/m9sweeper && \
helm repo update && \
helm upgrade m9sweeper m9sweeper/m9sweeper --install --wait --create-namespace --namespace m9sweeper-system \
  --values values.yaml --version latest

When implementing for our customers we automate this in a CICD pipeline. Upgrades can be done simply by changing which chart version you are deploying. By default, it installs the latest version, but you can add –version to install a specific version of m9sweeper.

At a minimum, you MUST specify these 4 values:

--set-string dash.init.superAdminEmail="super.admin@m9sweeper.io"
--set-string dash.init.superAdminPassword="password"
--set-string global.jwtSecret="changeme" \
--set-string global.apiKey="YOUR-API-KEY"

Validating Webhook

If you wish to have m9sweeper prevent applications from booting up that are not compliant with your specified policies, you will need the validating webhook. This installs automatically and should work without any configuration.

However, if you are running in Azure Kubernetes Service OR have the kubernetes api firewalled in such a way that it cannot reach out to a remote url for the validating webhook, then you will need to setup an nginx reverse proxy using our reverse proxy self-installer. This script will generate a CA Certificate Bundle and Nginx configuration to enable the reverse proxy to work in Azure Kubernetes Service.

Falco bulkhead Deployment

Depending on your environment, Falco can send an immense amount of alerts to M9sweeper. This could overload the application and cause it to crash. To avoid this, there is an option to deploy multiple instances of M9sweeper that only take api requests sent to /api/falco.

falco:
  bulkhead: true
  replicas: 1

When set to true, this will create a seperate deployment appended with “-falco”, a service, and when applicable, a service monitor. This also edits ingress, routing all API calls from Falco, into the bulkhead deployment.

Configuration Options

If postgresql is enabled, then it will deploy postgres db. Set to false to use an external postgres DB

postgres:
  enabled: true

If rabbitmq is enabled, then it will deploy rabbitmq. Set to false to use an external rabbitmq.

rabbitmq:
  enabled: true

The following table lists the configurable parameters of the chart and the default values.

4 - Concepts

4.1 - Image Scanning

Trivy is one of the best tools for scanning Kubernetes images, and m9sweeper can coordinate scanning images deployed to your cluster, rescanning of those images, as well as blocking images from deploying if they do not meet your minimum criteria for compliance.

M9sweeper also allows you to create exceptions or have your employees' request exceptions be approved when they do not have the time to fix an issue in the moment but still want to allow applications to deploy.

For a full list of trawler configuration options, see the trawler reference guide

4.1.1 - Automating Scanning

By default, m9sweeper will scan all images that it sees deployed in your cluster that have not been scanned.

Also, you can configure the Image Rescan Period (Days) when setting up policies to automatically rescan images. This will then rescan any images currently running in your cluster if they have not been scanned in a certain number of days.

4.1.2 - Scanning in CICD Pipelines

You can automatically scan images using trawler in your automated CICD pipelines. The easiest way to do this is by running trawler from the command line using the container image. It will look something like this.

docker run \
    --env "M9SWEEPER_URL=XXX" \
    --env "M9SWEEPER_API_KEY=XXX" \
    --env "CLUSTER_NAME=XXX" \
    --env "DOCKER_IMAGE_URL=XXX" \
    -it m9sweeper/trawler trawler scan

Note that you will need to provide an API key as well as the name of the cluster you are scanning it for so that it can authenticate with m9sweeper. You will have to run a scan for each cluster you plan to deploy it to because each cluster might have different policies setup.

4.1.3 - Enforcing Compliance

4.1.3.1 - Setting Up Policies

Policy Settings

In the organization settings, you can click on policies in the left navigation and configure one or more policies for your cluster. These policies define what criteria an image must meet to be considered compliant in the cluster.

It looks something like this.

Only policies and scanners that are active and required will be used in determing whether an image is compliant. Also, when evaluating an image for a cluster, only policies that are configured for that cluster will be applied.

Configuring Trivy Requirements

When configuring the trivy scanner, you can define the maximum number of vulnerabilities for each category. The defaults that come pre-installed essentially will block any image with a fixable major or critical vulnerability.

4.1.3.2 - Installing Webhook

In order to have m9sweeper enforce image scanning compliance in your cluster, you need to install a validating webhook in your cluster. This should be done automatically by m9sweeper during the setup process, but if for some reason it was not you can click “Update Kubeconfig” on your cluster’s settings page and run through the setup wizard again to have it install the webhook for you.

4.1.3.3 - Enabling Enforcement

To enable enforcement, you need to make your way to the Cluster settings for your cluster and check the box that enables webhook enforcement.

Once checked, anything that is not compliant with the policies you have setup will be prevented from deploying. Note that this only works if you have installed the webhook during the setup process.

4.1.3.4 - Exceptions

Sometimes, for practical reasons, you may need to allow something with a known security issue to continue to be deployed in an environment. You can do this using exceptions.

Creating Exceptions

Your team can create exceptions when the need arises.

Temporary Exceptions

When a new exception is discovered such as through a nightly image rescan, you may want automatically provide teams with a certain amount of time (lets say a week) before it would block their deployments. This can be done through the use of a temporary exception.

To enable this feature, you need to edit the policy that is setup for your cluster(s) and check the box (see below) and set how many days the temporary exceptions should be active.

When new temporary exceptions are created, it will email all of your admins to review and decide what to do. They should notify your software development teams if the issue should be resolved right away and/or change the end date on the exception.

Exception Statuses

Active: Active exceptions are the only exceptions that will be used when validating image compliance, and only if the current date is within the exception’s start and end date.

In Review: When an exception is submitted for review, it will be in this status. It will not be used when validating an image’s compliance, but someone should review to decide whether it is a risk your organization is willing to take.

Inactive: The exception will be ignored when validating image compliance.

Requesting Exceptions

When viewing an image, if a team member who is NOT an admin believes an exception is required, they can request an exception. This exception falls into the In Review status and will not be active, but it does provide a forum for your team to request exceptions and for someone else (such as your security/ops team) to review and approve the exception. They would approve the exception by changing its status to Active.

Exception Types

Exception types are available as options: Policy and Override.

• Policy: Allow listed policy(s) to be bypassed. When this type is chosen, user should also select scanner(s) and policy(s) for the desired exception.
• Override: When this type is chosen, user should select an alternate severity level for desired exception.

4.2 - kube-bench

kube-bench will run a scan of your cluster to compare its configuration against the Center for Internet Security (CIS) benchmarks. This is a good way to check for obvious configuration issues, such as allowing anonymous users. It deploys as an application in your cluster and then accesses the Kubernetes APIs to see how your cluster is configured.

We recommend setting up kube-bench to run as a nightly cron job so that you can see the effect of any changes you make to your cluster.

First, you need to install kube-bench and set it up to upload its results to m9sweeper. To do this, go to kube-bench for your cluster and click on Run Audit in the top right.

Then, you can use the wizard to generate a CLI command that will install kube-bench using our helm chart as a cron job or one-time job in your cluster. It will upload its results back to the API (and you should see an API key in the url).

Note that this will only work IF you have enabled traffic ingressing or otherwise allowed kube-bench to pipe its results back to the m9sweeper dash app.

And then this will display a summary report, like this:

You can click on any line to expand it and see directions for remediation.

4.3 - kube-hunter

kube-hunter will run a non-invasive (or invasive, if you want) penetration test of your cluster. It deploys as an application in your cluster and then attempts to explore and see what all it is able to do. It reports back on any concerns that you should be aware of.

We recommend setting up kube-hunter to run as a nightly cron job so that you can see the effect of any changes you make to your cluster.

First, you need to install kube-hunter and set it up to upload its results to m9sweeper. To do this, go to kube-hunter for your cluster and click on Run Audit in the top right.

Then, you can use the wizard to generate a CLI command that will install kube-hunter using our helm chart as a cron job or one-time job in your cluster. It will upload its results back to the API (and you should see an API key in the url).

Note that this will only work IF you have enabled traffic ingressing or otherwise allowed kube-hunter to pipe its results back to the m9sweeper dash app.

And then this will display a summary report, like this:

4.4 - KubeSec

KubeSec coaches you about how to make your deployments more secure. You can find it in the left navigation after selecting a cluster.

To get started, select a pod you want to evaluation and click Run KubeSec.

It will then display a report of your pods' compliance and any improvements that could be made.

4.5 - Gatekeeper

4.5.1 - Getting Started

Gatekeeper is a great tool for creating rules for your Kubernetes cluster. You configure rules using constraint templates and constraints.

Constraint templates, such as this, allow you to define rules using a language called Rego.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Constraints are then created to apply these rules to a particular set of kubernetes entities. These constraints can also contain configuration parameters, such as which labels are required (in this example).

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Namespace"]
  parameters:
    labels: ["gatekeeper"]

M9sweeper makes managing your constraint templates and constraints extremely convenient through an easy-to-use user interface.

To get started, you need to install Gatekeeper.

Read more about constraint templates

4.5.2 - Managing Constraints

Managing Constraint Templates

After installing Gatekeeper, your next step is to install constraint templates. You can do this using a CICD pipeline, or if you are new to this you can use the m9sweeper graphical user interface to install Constraint Templates from our library of templates.

First, open the Gatekeeper page for your cluster and click on “+Add More” in the top right.

Then, check the boxes on the constraint templates you want to install and click save changes.

After doing this, you will see the list of constraint templates has been installed.

Managing Constraints

Just installing constraint templates alone does not do anything - you also have to apply these constraint templates to specific workloads / namespaces. This is done through the use of constraints.

If you click on one of your constraint templates, you will be taken to a page that lists all of the constraints created for this constraint template. After doing so, click “+Add More” to create a constraint for this template.

In this user interface, it will let you set which namespaces and type of entity it applies to. Reasonable defaults are typically filled in if you used one of our templates.

If properties can be configured, we will automatically generate a user interface for configuring those properties. You can fill in the required properties.

Do not forget to select whether it is to be in enforcement mode or audit mode. Only enforcement mode is actually enforced - audit mode is purely used for evaluation purposes.

Click save changes and now you should have your constraint created!

4.5.3 - Reviewing Exceptions

While gatekeeper constraints can be scoped to specific namespaces or entity types, sometimes you want to create temporary exceptions for a particular namespace that end at a specific date, or sometimes you want to just target a specific workload.

In those cases, you can use our exceptions feature. Our exceptions feature will automatically code-generate rego code nightly, re-evaluating the exceptions every day and taking into account the exception’s status, start date, and end date. These exceptions work just like image compliance exceptions, except they target gatekeeper constraint templates rather than image scanning rules.

To use them, be sure and pick Gatekeeper as the exception type:

4.5.4 - Reviewing Pod Compliance

To view a pod’s compliance, navigate to your cluster’s list of workloads. Then, click on the namespace you want to review.

Next, it will list all pods in the namespace. Note that it re-populates this list hourly.

If you click on a pod, it will list all images in the pod and those images' own compliance.

If you click on the Gatekeeper icon in the top right, it will tell if you if any violations exist and, if so, what violations exist.

4.6 - Falco

What is Falco?

The Falco Project is an open source runtime security tool originally built by Sysdig, Inc.

What does Falco do?

Falco uses system calls to secure and monitor a system, by:

• Parsing the Linux system calls from the kernel at runtime
• Asserting the stream against a powerful rules engine
• Alerting when a rule is violated

For more information, see the Falco Documentation.

Setup and configuration

M9sweeper consumes HTTP requests from Falco in JSON format to present readable information in our UI.

To accomplish this, FalcoSideKick is deployed to give us more control over Falco’s output.

Configuration Notes:

• Set the config.webhook.address value to your instance of M9sweeper.
• Depending on how you are deploying M9sweeper, you might need to set config.webhook.checkcert=false.
• We recommend setting the minimum priority to “error”. For more information see Minimum Priority

Deploy Falco

Add the Helm Chart Repo then install Falco:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
 --create-namespace \
 --namespace falco \
 --set falco.driver.enabled=true \
 --set-string falco.driver.kind=ebpf \
 --set falco.tty=true \
 --set falco.json_output=true \
 --set falco.json_include_output_property=true \
 --set falco.http_output.enabled=true \
 --set-string falco.http_output.url=http://falcosidekick:2801/ \
 --set falcosidekick.enabled=true \
 --set-string falcosidekick.config.webhook.address= 'https://M9SWEEPER/api/falco/CLUSTERID/create/?key=FALCO_API_KEY' \
 --set falcosidekick.config.webhook.checkcert=true \
 --set-string falcosidekick.config.webhook.minimumpriority='error'

Notes:

• Make sure to change the URL value to point to your M9sweeper instance as well as enter the CLUSTER_ID of whatever cluster it is supposed to save to.

• We recommend using the EBPF driver, however, if you have issues please refer to to the installation page. Or you may try to use the kernel driver and set “falco.driver.kind=module” above.

Anomaly Detection:

• Anomaly instrusion detection plays vital role in protecting networks against malicious activities.
• It detects unusual behaviors or threats in cloud-native environments with about 100 out-of-the-box security rules.
• Administrators can set up alert to receive email notification based on specific priority (severity) level(s) and email frequency that fits individual application needs.
• Simply go to Falco > Settings

Here is an example of Non-authorized container namespace change violation:

rule: change_thread_namespace
desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns.
condition: syscall.type = setns and not proc.name in (docker, sysdig, dragent)
output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)"
priority: WARNING

Minimum Priority

Falco Alerts can get very noisy. This option lets you choose which levels of alerts you want to include/exclude. Once set, all rules having a priority more severe than this level will be loaded/run. The default is an empty string and includes all alerts. The order is as follows:

• Emergency
• Alert
• Critical
• Error
• Warning
• Notice
• Info
• Debug

For more information on Priority please refer to Falco Docs - Priorities

5 - Reference

The app consists of several components. Each is documented below.

5.1 - Dash Parameters

The most common way to configure dash is to simply configure environment variables. Some parameters are only used during the initial installation.

For details about configuring environment variables for the dash app, see the advanced install guide.

Required Parameters

Complete List of ENV Variables

Server Configuration

Database Configuration

Email Configuration

RabbitMQ Configuration

File Storage Configuration

Misc App Configuration

5.2 - Trawler Parameters

Trawler is our app for running scans of your containers. Right now, it is a wrapper around Trivy, one of the best container scanners available, but it is architected such that other scanners could be supported in the future (contributions are welcome!).

Functions

Trawler is normally run in listening mode in the cluster and in scanning mode when scanning a single image from a CICD Pipeline.

You can see how to use one mode or the other by just viewing the built-in help docs.

trawler help

The output looks something like this:

Usage:
trawler [-Dh] [-A=<m9sweeperApiKey>] [-H=<rabbitmqHostname>]
[-p=<rabbitmqPassword>] [-P=<trawlerParallelScanners>]
[-q=<rabbitmqQueueName>] [-t=<rabbitmqPort>] [-u=<rabbitmqUsername>]
[-U=<m9sweeperUrl>] [COMMAND]

Description:
Run Trawler in its RabbitMQ mode where it will monitor a specified RabbitMQ
queue for scan jobs.

Options:
-P, --parallel-scans=<trawlerParallelScanners>
number of scanners that Trawler can run at once
-U, --url=<m9sweeperUrl>   URL of the m9sweeper instance
-A, --api-key=<m9sweeperApiKey>
API Key of the m9sweeper instance
-D, --debug                whether to enable debug logs
-u, --rabbitmq-user=<rabbitmqUsername>
username of the RabbitMQ server
-p, --rabbitmq-password=<rabbitmqPassword>
password of the RabbitMQ server
-H, --rabbitmq-host=<rabbitmqHostname>
hostname of the RabbitMQ server
-t, --rabbitmq-port=<rabbitmqPort>
port of the RabbitMQ server
-q, --rabbitmq-queue=<rabbitmqQueueName>
name of the RabbitMQ queue to listen on
-h, --help                 display this help and exit

Commands:
scan  Scan a single docker image in the standalone scan mode.

Many settings can be set through the CLI.

Running an image scan is as simple as:

trawler scan alpine:3.15

Environment Variables

In addition to CLI parameters, you can also configure Trawler using environment variables. This is the norm when deploying a trawler runner to run automatic scans of new, unrecognized images as well as nightly image scans.

General Configuration Options

Configuration Options for Running a Scan Worker

Configuration Options for Scanning a Single Image

5.3 - API Docs

API documentation is provided through swagger, which is available at /doc/. For example, if you are hosting it at m9s.yoursite.com, it would be available at https://m9s.yoursite.com/doc/

6 - Contributing to m9sweeper

Contributing to m9sweeper

DEVELOPING

In our GitHub repository, our CONTRIBUTING.md contains all the steps to get started with hacking m9sweeper.

We are very excited to have you a part of our community!

REPOSITORIES

There are multiple repositories that comprise the m9sweeper platform @TODO - create repos in github and link them here

m9sweeper Repo: This repo is the main repo that integrates all the other repos together.

Dash Repo: This repo contains the dash application and all the jobs/scripting to install the platform

Trawler Repo: This repo is where the scanning functionality of the platform is developed.

Kube Hunter Repo: Our helm wrapper for the kube-hunter image that is integrated in the m9sweeper platform

Kube Bench Repo: Our helm wrapper for the kube-bench image that is integrated in the m9sweeper platform

BUGS If you find any bugs or are having any trouble, please contact us by filing an issue in the m9sweeper repo issues.

If you have any updates to our documentation, please make any PRs to our docs repo or click on the Edit this page to be taken directly to the page.