Concepts

• 1: Image Scanning
• 1.1: Automating Scanning
• 1.2: Scanning in CICD Pipelines
• 1.3: Enforcing Compliance
• 1.3.1: Setting Up Policies
• 1.3.2: Installing Webhook
• 1.3.3: Enabling Enforcement
• 1.3.4: Exceptions
• 2: kube-bench
• 3: kube-hunter
• 4: KubeSec
• 5: Gatekeeper
• 5.1: Getting Started
• 5.2: Managing Constraints
• 5.3: Reviewing Exceptions
• 5.4: Reviewing Pod Compliance
• 6: Falco
• 7: Single Sign-On
• 7.1: OAuth2 Login
• 7.2: LDAP Login
• 8:
• 8.1:

1 - Image Scanning

Trivy is one of the best tools for scanning Kubernetes images, and m9sweeper can coordinate scanning images deployed to your cluster, rescanning of those images, as well as blocking images from deploying if they do not meet your minimum criteria for compliance.

M9sweeper also allows you to create exceptions or have your employees' request exceptions be approved when they do not have the time to fix an issue in the moment but still want to allow applications to deploy.

For a full list of trawler configuration options, see the trawler reference guide

1.1 - Automating Scanning

By default, m9sweeper will scan all images that it sees deployed in your cluster that have not been scanned.

Also, you can configure the Image Rescan Period (Days) when setting up policies to automatically rescan images. This will then rescan any images currently running in your cluster if they have not been scanned in a certain number of days.

1.2 - Scanning in CICD Pipelines

You can automatically scan images using trawler in your automated CICD pipelines. The easiest way to do this is by running trawler from the command line using the container image. It will look something like this.

docker run \
    --env "M9SWEEPER_URL=XXX" \
    --env "M9SWEEPER_API_KEY=XXX" \
    --env "CLUSTER_NAME=XXX" \
    --env "DOCKER_IMAGE_URL=XXX" \
    -it m9sweeper/trawler trawler scan

Note that you will need to provide an API key as well as the name of the cluster you are scanning it for so that it can authenticate with m9sweeper. You will have to run a scan for each cluster you plan to deploy it to because each cluster might have different policies setup.

1.3 - Enforcing Compliance

1.3.1 - Setting Up Policies

Policy Settings

In the organization settings, you can click on policies in the left navigation and configure one or more policies for your cluster. These policies define what criteria an image must meet to be considered compliant in the cluster.

It looks something like this.

Only policies and scanners that are active and required will be used in determing whether an image is compliant. Also, when evaluating an image for a cluster, only policies that are configured for that cluster will be applied.

Configuring Trivy Requirements

When configuring the trivy scanner, you can define the maximum number of vulnerabilities for each category. The defaults that come pre-installed essentially will block any image with a fixable major or critical vulnerability.

1.3.2 - Installing Webhook

In order to have m9sweeper enforce image scanning compliance in your cluster, you need to install a validating webhook in your cluster. This should be done automatically by m9sweeper during the setup process, but if for some reason it was not you can click “Update Kubeconfig” on your cluster’s settings page and run through the setup wizard again to have it install the webhook for you.

1.3.3 - Enabling Enforcement

To enable enforcement, you need to make your way to the Cluster settings for your cluster and check the box that enables webhook enforcement.

Once checked, anything that is not compliant with the policies you have setup will be prevented from deploying. Note that this only works if you have installed the webhook during the setup process.

1.3.4 - Exceptions

Sometimes, for practical reasons, you may need to allow something with a known security issue to continue to be deployed in an environment. You can do this using exceptions.

Creating Exceptions

Your team can create exceptions when the need arises.

Temporary Exceptions

When a new exception is discovered such as through a nightly image rescan, you may want automatically provide teams with a certain amount of time (lets say a week) before it would block their deployments. This can be done through the use of a temporary exception.

To enable this feature, you need to edit the policy that is setup for your cluster(s) and check the box (see below) and set how many days the temporary exceptions should be active.

When new temporary exceptions are created, it will email all of your admins to review and decide what to do. They should notify your software development teams if the issue should be resolved right away and/or change the end date on the exception.

Exception Statuses

Active: Active exceptions are the only exceptions that will be used when validating image compliance, and only if the current date is within the exception’s start and end date.

In Review: When an exception is submitted for review, it will be in this status. It will not be used when validating an image’s compliance, but someone should review to decide whether it is a risk your organization is willing to take.

Inactive: The exception will be ignored when validating image compliance.

Requesting Exceptions

When viewing an image, if a team member who is NOT an admin believes an exception is required, they can request an exception. This exception falls into the In Review status and will not be active, but it does provide a forum for your team to request exceptions and for someone else (such as your security/ops team) to review and approve the exception. They would approve the exception by changing its status to Active.

Exception Types

Exception types are available as options: Policy and Override.

• Policy: Allow listed policy(s) to be bypassed. When this type is chosen, user should also select scanner(s) and policy(s) for the desired exception.
• Override: When this type is chosen, user should select an alternate severity level for desired exception.

2 - kube-bench

kube-bench will run a scan of your cluster to compare its configuration against the Center for Internet Security (CIS) benchmarks. This is a good way to check for obvious configuration issues, such as allowing anonymous users. It deploys as an application in your cluster and then accesses the Kubernetes APIs to see how your cluster is configured.

We recommend setting up kube-bench to run as a nightly cron job so that you can see the effect of any changes you make to your cluster.

First, you need to install kube-bench and set it up to upload its results to m9sweeper. To do this, go to kube-bench for your cluster and click on Run Audit in the top right.

Then, you can use the wizard to generate a CLI command that will install kube-bench using our helm chart as a cron job or one-time job in your cluster. It will upload its results back to the API (and you should see an API key in the url).

Note that this will only work IF you have enabled traffic ingressing or otherwise allowed kube-bench to pipe its results back to the m9sweeper dash app.

And then this will display a summary report, like this:

You can click on any line to expand it and see directions for remediation.

3 - kube-hunter

kube-hunter will run a non-invasive (or invasive, if you want) penetration test of your cluster. It deploys as an application in your cluster and then attempts to explore and see what all it is able to do. It reports back on any concerns that you should be aware of.

We recommend setting up kube-hunter to run as a nightly cron job so that you can see the effect of any changes you make to your cluster.

First, you need to install kube-hunter and set it up to upload its results to m9sweeper. To do this, go to kube-hunter for your cluster and click on Run Audit in the top right.

Then, you can use the wizard to generate a CLI command that will install kube-hunter using our helm chart as a cron job or one-time job in your cluster. It will upload its results back to the API (and you should see an API key in the url).

Note that this will only work IF you have enabled traffic ingressing or otherwise allowed kube-hunter to pipe its results back to the m9sweeper dash app.

And then this will display a summary report, like this:

4 - KubeSec

KubeSec coaches you about how to make your deployments more secure. You can find it in the left navigation after selecting a cluster.

To get started, select a pod you want to evaluation and click Run KubeSec.

It will then display a report of your pods' compliance and any improvements that could be made.

5 - Gatekeeper

5.1 - Getting Started

Gatekeeper is a great tool for creating rules for your Kubernetes cluster. You configure rules using constraint templates and constraints.

Constraint templates, such as this, allow you to define rules using a language called Rego.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Constraints are then created to apply these rules to a particular set of kubernetes entities. These constraints can also contain configuration parameters, such as which labels are required (in this example).

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Namespace"]
  parameters:
    labels: ["gatekeeper"]

M9sweeper makes managing your constraint templates and constraints extremely convenient through an easy-to-use user interface.

To get started, you need to install Gatekeeper.

Read more about constraint templates

5.2 - Managing Constraints

Managing Constraint Templates

After installing Gatekeeper, your next step is to install constraint templates. You can do this using a CICD pipeline, or if you are new to this you can use the m9sweeper graphical user interface to install Constraint Templates from our library of templates.

First, open the Gatekeeper page for your cluster and click on “+Add More” in the top right.

Then, check the boxes on the constraint templates you want to install and click save changes.

After doing this, you will see the list of constraint templates has been installed.

Managing Constraints

Just installing constraint templates alone does not do anything - you also have to apply these constraint templates to specific workloads / namespaces. This is done through the use of constraints.

If you click on one of your constraint templates, you will be taken to a page that lists all of the constraints created for this constraint template. After doing so, click “+Add More” to create a constraint for this template.

In this user interface, it will let you set which namespaces and type of entity it applies to. Reasonable defaults are typically filled in if you used one of our templates.

If properties can be configured, we will automatically generate a user interface for configuring those properties. You can fill in the required properties.

Do not forget to select whether it is to be in enforcement mode or audit mode. Only enforcement mode is actually enforced - audit mode is purely used for evaluation purposes.

Click save changes and now you should have your constraint created!

5.3 - Reviewing Exceptions

While Gatekeeper constraints can be scoped to specific namespaces or entity types, sometimes you want to create temporary exceptions for a particular namespace that end at a specific date, or sometimes you want to just target a specific workload.

In those cases, you can use our exceptions feature. Our exceptions feature will automatically code-generate rego code nightly, re-evaluating the exceptions every day and taking into account the exception’s status, start date, and end date. These exceptions work just like image compliance exceptions, except they target Gatekeeper constraint templates rather than image scanning rules.

To use them, be sure and pick Gatekeeper as the exception type:

5.4 - Reviewing Pod Compliance

To view a pod’s compliance, navigate to your cluster’s list of workloads. Then, click on the namespace you want to review.

Next, it will list all pods in the namespace. Note that it re-populates this list hourly.

If you click on a pod, it will list all images in the pod and those images' own compliance.

If you click on the Gatekeeper icon in the top right, it will tell if you if any violations exist and, if so, what violations exist.

6 - Falco

What is Falco?

The Falco Project is an open source runtime security tool originally built by Sysdig, Inc.

What does Falco do?

Falco uses system calls to secure and monitor a system, by:

• Parsing the Linux system calls from the kernel at runtime
• Asserting the stream against a powerful rules engine
• Alerting when a rule is violated

For more information, see the Falco Documentation.

Setup and configuration

M9sweeper consumes HTTP requests from Falco in JSON format to present readable information in our UI.

To accomplish this, FalcoSideKick is deployed to give us more control over Falco’s output.

Configuration Notes:

• Set the config.webhook.address value to your instance of M9sweeper.
• Depending on how you are deploying M9sweeper, you might need to set config.webhook.checkcert=false.
• We recommend setting the minimum priority to “error”. For more information see Minimum Priority

Deploy Falco

Add the Helm Chart Repo then install Falco:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --wait \
  --create-namespace \
  --namespace falco \
  --set falco.driver.enabled=true \
  --set-string falco.driver.kind=ebpf \
  --set falco.tty=true \
  --set falco.json_output=true \
  --set falco.json_include_output_property=true \
  --set falco.http_output.enabled=true \
  --set-string falco.http_output.url=http://falcosidekick:2801/ \
  --set falcosidekick.enabled=true \
  --set-string falcosidekick.config.webhook.address= 'https://M9SWEEPER/api/falco/CLUSTERID/create/?key=FALCO_API_KEY' \
  --set falcosidekick.config.webhook.checkcert=true \
  --set-string falcosidekick.config.webhook.minimumpriority='error'

Notes:

• Make sure to change the URL value to point to your M9sweeper instance as well as enter the CLUSTER_ID of whatever cluster it is supposed to save to.

• We recommend using the EBPF driver, however, if you have issues please refer to to the installation page. Or you may try to use the kernel driver and set “falco.driver.kind=module” above.

Anomaly Detection:

• Anomaly instrusion detection plays vital role in protecting networks against malicious activities.
• It detects unusual behaviors or threats in cloud-native environments with about 100 out-of-the-box security rules.
• Administrators can set up alert to receive email notification based on specific priority (severity) level(s) and email frequency that fits individual application needs.
• Simply go to Falco > Settings

Here is an example of Non-authorized container namespace change violation:

rule: change_thread_namespace
desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns.
condition: syscall.type = setns and not proc.name in (docker, sysdig, dragent)
output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)"
priority: WARNING

Minimum Priority

Falco Alerts can get very noisy. This option lets you choose which levels of alerts you want to include/exclude. Once set, all rules having a priority more severe than this level will be loaded/run. The default is an empty string and includes all alerts. The order is as follows:

• Emergency
• Alert
• Critical
• Error
• Warning
• Notice
• Info
• Debug

For more information on Priority please refer to Falco Docs - Priorities

7 - Single Sign-On

M9sweeper supports user configurable single sign-on using both OAuth2 and LDAP.

7.1 - OAuth2 Login

M9sweeper supports using OAuth2 to authenticate, and has been tested with Google and Microsoft Azure AD. To add one of these login methods, go to “Organization Settings” > “Sign on Methods” and select “Add External Auth Configuration.” Then select the OAUTH2 Auth Type. After, select your desired implementation from the Provider Type dropdown, with implementation details found below.

Make sure to give the sign-on method a unique name. Note that the Auth Name selected will be what users see when selecting a login method.

Google Oauth2

Using Google OAuth requires first setting up credentials through Google’s cloud platform console. Instructions on setting up necessary credentials can be found through Google’s official docs. When setting up credentials for M9sweeper, be sure to select the “Web Application” application type.

After creating your credentials, you will need to take the Client ID and Client Secret values generated by Google and add them to their respective fields in the configuration form.

Access Token Uri and Authorization Uri

These fields refer to the endpoints which the app will reach out to during the authorization process. They are automatically filled with the default values used by Google, and you should not need to modify them yourself.

Access Scopes

A comma separated list of the fields the app will request access to when querying data from a user. This is automatically filled with the information M9sweeper needs to access, and you should not need to modify it yourself.

Allowed User Domains

A comma separated list of what email domains are allowed access to your M9sweeper instance. This field requires the full domain name following the “@” symbol in your email addresses: for example, enter “example.com” to allow “any.user@example.com” to access your M9sweeper instance.

When a user first logs in using OAuth, if they are from an allowed domain they will be granted Read Only access, which you can freely change later on.

If someone attempts to log in using OAuth from a domain not in the allowed list, they will receive an “Access Denied” error.

Microsoft Azure OAuth2

Using Azure OAuth first requires setting up credentials through Microsoft’s Azure portal. Instructions on how to create these credentials can be found in Microsoft’s official docs.

After setting up credentials through Azure, you will need the Client ID for configuration in M9sweeper, added in the Client ID field of the configuration form.

Authorization Uri

The endpoint that M9sweeper will use to authenticate with Microsoft’s OAuth system. This field is automatically filled with the default endpoint, and you should not need to modify it yourself.

Access Scopes

A comma separated list of fields that M9sweeper’s OAuth will request access to. This field is automatically filled and should not need further modification.

Allowed User Domains

A comma separated list of what email domains are allowed access to your M9sweeper instance. This field requires the full domain name following the “@” symbol in your email addresses: for example, enter “example.com” to allow “any.user@example.com” to access your M9sweeper instance.

When a user first logs in using OAuth, if they are from an allowed domain they will be granted Read Only access, which you can freely change later on.

If someone attempts to log in using OAuth from a domain not in the allowed list, they will receive an “Access Denied” error.

7.2 - LDAP Login

M9sweeper supports user login using LDAP with a configurable implementation. Setting up this login method requires some knowledge of the underlying directory structure, which will be specific to your LDAP implementation.

Initial Configuration Setup

To begin setting up an LDAP server, log into M9sweeper as an admin and go to “Organization Settings” > “Sign on Methods”, then select “Add External Auth Configuration.” From the “Auth Type” dropdown, select “LDAP” to open the configuration menu.

Provider Type

Identifies the type of LDAP system being used to authenticate users. Any value can be entered, to match the LDAP system in use by your organization.

Auth Name

A unique name to identify the sign on method; this will be what users see when selecting a method to log in with.

URL

The URL where the LDAP server can be reached. It should be defined with ldap: as the protocol, such as ldap://example.org

User Search Base

The base Distinguished Name (DN) at which users can be found, consisting of all the ancestor entries of a user entry; so if a specific user would be defined as cn=exampleuser,dc=example,dc=org, the user search base would be dc=example,dc=org. Alternatively, a user defined as uid=anotheruser,ou=users,dc=example,dc=org would have a search base of ou=users,dc=example,dc=org.

Username Attribute

The attribute used to define unique users found at the user search base, this is the field M9sweeper will search by when given a username. For the user cn=exampleuser,dc=example,dc=org the username attribute would be cn, and for uid=anotheruser,ou=users,dc=example,dc=org the attribute would be uid.

Admin DN and Admin Password

The full DN an admin account on the server, along with its password. In order to pull the necessary information from an LDAP server, M9sweeper needs to connect with admin credentials, which are provided here. The admin DN is the full DN of a given user, such as cn=admin,dc=example,dc=org.

Default User Authority

The default permissions given to a user who has a valid account in the LDAP server, but is not part of a group with specific permissions (which will be defined in the following steps). Choosing an authority other than None will allow any member of the organization to log into M9sweeper, with whatever rights are defined. Users part of a group with defined permissions will have those permissions override whatever is set here, even if those permissions are lower.

Setting the default authority as None will cause users that are not part of a group with access permissions to be given an “Access Denied” error upon logging in.

Group User Authority Settings

The following steps go over how to manage user authority through LDAP groups. These steps are optional, in which case you will need to set a default user authority for all users logging in with LDAP, however it is much more secure to define access controls by group.

Group Search Base

The base DN that M9sweeper should look for groups at, defined the same way as the user search base; a group such as cn=usergroup,ou=groups,dc=example,dc=org would have a search base of ou=groups,dc=example,dc=org.

Group Class

The object class that groups are instances of in your LDAP server, such as posixGroup. If not known, you can find this value by searching for groups in your LDAP server and looking for the objectClass attribute of your user groups.

Group Member Attribute

The attribute within a group that defines which users are members of that group. A group defined as follows:

dn: cn=readonlyusers,ou=groups,dc=example,dc=org
memberUid: exampleuser
memberUid: anotheruser
cn: readonlyusers

Referencing users defined as:

dn: uid=exampleuser,dc=example,dc=org
uid: exampleuser

dn: uid=anotheruser,dc=example,dc=org
uid: anotheruser

Would use memberUid as the Group Member Attribute.

Group Member User Attribute

The unique attribute of users which groups reference to establish membership. A group defined as follows:

dn: cn=readonlyusers,ou=groups,dc=example,dc=org
memberUid: exampleuser
memberUid: anotheruser
cn: readonlyusers

Referencing users defined as:

dn: uid=exampleuser,dc=example,dc=org
uid: exampleuser

dn: uid=anotheruser,dc=example,dc=org
uid: anotheruser

Would use uid as the Group Member User Attribute.

Group Auth Level Attribute

The attribute of a group used to define the authority level of a group. Any attribute can be selected, though selecting a unique attribute allows for tighter control of M9sweeper permissions.

Group Auth View Only Value, Group Auth Admin Value, and Group Auth Super Admin Value

These define which values of the previous attribute correspond to which auth level in M9sweeper.

8 -

8.1 -