Using Trivy to Scan Docker Images (and how to patch the vulnerabilities)

Welcome back for the next installment of our security newsletter! I hope you enjoyed our last article on 3 Kubernetes Security Tools in 3 Minutes.

In this article, we are going to give you some boots-on-the-ground guidance on the use of Trivy to perform a scan as well as how to actually go about patching any issues that Trivy has found.

Using Trivy to Run a Scan

You will first want to start trivy. If you are on linux or mac, you can install it using many standard package managers.

Another option is to use docker to run trivy on your local machine, such as:

docker run --rm -v $HOME/Library/Caches:/root/.cache/ aquasec/trivy python:3.4-alpine

More examples are available on Trivy’s Dockerhub page.

Getting All the Output from Trivy

Trivy, by default, formats the output in a nice-easy to read format. However, sometimes not all of the details are outputted this way. To get more information, you may need to have Trivy dump out the report in another format, such as JSON or HTML. At the end of the day, the JSON is the complete raw data output of a Trivy scan, and all the other templates are just formatted to be easier to read in one way or another.

Okay, I ran a scan. What do I do next?

Trivy detects 3 types of vulnerabilities, broadly, and how you patch each is a bit different:

Operating System Package CVEs: Known vulnerabilities in operating system packages ought to be patched or replaced. This usually consists of upgrading to the latest operating system build (ex: latest alpine or debian version) and then doing an upgrade of the package. Sometimes, you may have to install the package manually or from an “edge” repository if the operating system’s repository maintainers have not yet pulled in the latest, patched versions of the package. Trivy will tell you if a fix is available and, if you expand the website it provides, it will tell you what version(s) have been patched.
Software/Code Library CVEs: Known vulnerabilities on javascript libraries, java packages, dotnet packages, ruby packages, and the like will be reported. You generally need to replace the packages with newer, patched versions, where possible. Trivy will tell you if a fix is available and, if you expand the website it provides, it will tell you what version(s) have been patched.
Misconfiguration: Trivy can tell you if you are running with escalated privileges (such as running as root) or if suspicious files (such as certificates/passwords) appear in the image. We provide more details on securing images below. You should also externalize certificates/passwords and inject them in as files/environment variables (or better yet, load from a secure vault such as hashicorp vault at runtime) so that if someone steals your container image, they cannot break into your system.

Securing Your Container Images

Fixing CVEs

Fixing CVEs primarily means upgrading to the latest (or at least newer) operating system packages and software libraries that no longer have the vulnerability. Keep in mind Semantic Versioning and how (in theory) minor or patch updates are generally backwards compatible, so upgrading to newer versions of those is generally very safe. Most modern packages (ahem, not dotnet though) follow semver naming conventions and as long as you are sticking to mainstream frameworks (React, VueJS, Angular, Ruby on Rails, Java Spring, NestJS, etc) you probably will not encounter many issues upgrading.

In our experience, the upgrade is not all that hard, but the testing is always the most tedious because upgrading to newer versions of packages you rely on can require full regression tests of your app. Things like automated testing and Blue/Green Deployments can help you move faster, but we do not want to underrate the importance of careful testing nor the amount of time that must be spent when you upgrade to newer versions of the software/framework that you rely on. Nonetheless, if a vulnerability warrants it, we strongly recommend you upgrade.

Not Running as Root

You really should never be giving your software root access unless it is absolutely necessary to do so. Software running as root can very easily escalate its privileges, breaking out of the container and breaking into other containers, ruining any kind of security plans you have in place. It can install packet sniffers, man in the middle attacks, dump and examine the raw memory of running processes, or install any software it can connect to download. Your software should not be running as root.

So how do you fix it? It is really easy to do in a dockerfile, actually. You generally just need to create a user/group and grant it access to execute whatever files need to be executed when the app boots up (but ideally, it would NOT be given write access to much of anything).

Something like this works:

# Create non root user
RUN addgroup -g 1050 -S limitedaccessaccount && \
    adduser -u 1050 -S limitedaccessaccount -G limitedaccessaccount
 
# Tell docker to run as non-root user
USER 1050

Limiting Access to Disk

While building your docker container, you should install all operating system packages and copy in all compiled code (or uncompiled code) in as root, initially, then grant the user your app will run as read+execute privileges over the code it needs to be able to run (and ideally nothing more than that). This ensures that even if the code is compromised, it can still run.

# Grant read+execute to the app directory (but NOT write)
RUN chmod -R +rx limitedaccessaccount:limitedaccessaccount /app

Do not Install More than you Need

Does your app really need git to run? Does it have apache left over, but it is not even used? Do you need those dev ruby on rails dependencies? Limit your attack surface by simply having less in your image. Just install what you actually will use.

Use base images like a debian slim image or alpine image to start that have very little on them. Only install the operating system and software dependencies you actually need to run, that way there is a lot less to compromise. Also, as a side benefit, your images will be smaller, boot up faster, and use less ram and storage at runtime.

How M9sweeper Can Help

M9sweeper runs Trivy and extends the functionality of Trivy in a variety of ways:

M9sweeper will scrape and track what is (and was previously) running in Kubernetes
M9sweeper then automatically scans with Trivy using our Trawler service. We recommend putting this in your CICD pipelines so that you can catch it before even deploying to Kubernetes.
M9sweeper lets you define policy around how many issues of which severity are considered compliant or not compliant.
M9sweeper allows you to create exceptions when you have determined that an issue is not exploitable or are willing to accept a risk. We also have an approval system where developers can request an exception and then you can discuss it on the platform. Exceptions can also have time limits.
M9sweeper can (optionally) block applications from booting up that do not meet your policy requirements (such as those with fixable, critical vulnerabilities).

Conclusion

We regularly get brought into projects to assist in securing an application. If it is built with Docker, we start with Trivy to figure out what obvious vulnerabilities exist, and follow the road map you see here to fix each issue. Then, move on to digging through the application itself for major OWASP vulnerabilities or concerning architectural patterns.