Capstone Project: Week 12, Security Testing

Hi everyone, welcome to Week 12, one of the last weeks of my capstone project: CI/CD Pipelines. I believe I have got a lot done this week in terms of meeting my stretch goal of incorporating DevSecOps principles such as security and secret scanning.

I am very excited to share with you what I have done this week, so let's get started!

Outcomes

• Docker Issue Fixed - I have fixed the issue with my unit test in my pipeline mentioned in my last post where I couldn't start my Docker container locally; no need for the work around.
• Talisman Pre-Commit - Using git pre-commit and talisman, I incorporated secret-scanning into my infrastructure before secrets can get to the online repository.
• Nikto - I incorporated nikto web server scanner into my pipeline to test my container locally as my first DAST integration.
• OWASP ZAP - I incorporated the renowned OWASP ZAP which is a web app vulnerability scanner like nikto that scans my staging environment before allowing release to production.
• Datadog Gates - I incorporated Datadog further into my DevOps Infrastructure solution by integrating Datadog monitor queries into my release pipeline which, if it report back alerts, the release will not be distributed to my production environment.

Fixing Docker for Unit Tests

As mentioned in my previous post, I was unable to build and run the docker image locally to perform my unit test using cURL. However, when I was doing more research on tests in Azure DevOps, I found the following documentation here

This documentation actually outlined how to both build and run a container locally. Based on what I did previously and what is demonstrated there, it seems it was my own mistake as I did not include the repository name in the command. Even though the repository is not on dockerhub or anywhere, it seems Azure DevOps still desired it. As such, I used the following script to get Azure DevOps to run the container locally without having to push it to a repository:

1- script: |
2    docker build -f $(Build.SourcesDirectory)/Dockerfile -t thylaw/testcontainer:$(tag) .
3    docker run --name flasktest --rm -d -p 5000:5000 thylaw/testcontainer:$(tag)    
4displayName: Build&Run Container

I am happy I discovered this as it removed a significant amount of time from my pipeline as well as eliminated a point of failure.

Secret Scanning with Talisman

One of the most important aspects of DevSecOps is the idea of "Secrets Management," where your secrets (such as API Keys) are obfuscated in code and use so that they can not be stolen and are only accessed when needed via secure channels. However, Secrets Management does not work if a developer accidentally pushes code to a repository without doing it the properly way, such as storing an API key directly in code. (I definitely have not done that before!)

As such, DevOps teams make great use of "Secret Scanning tools to find and remedy secrets within code or prevent them from being pushed altogether. I decided to incorporate secret scanning into my DevOps infrastructure so I went ahead and investigated tools such as Trufflehog.

Trufflehog was really cool and actually allows the user to scan not only code the code active within a repository, but all previous commits which can be accessed, because even if you had removed code from one commit or release, it may still be there on previous commits. I then looked at Detect-secrets but it didn't seem to detect my key and instead of attempting to configure it more, I decided to try out talisman first.

However, I realised this was a problem as it did nothing for preventing secrets from getting on the repository in the first place which really provides no purpose when used in a CI/CD environment. As such, I investigated more and found out about git pre-commit and talisman.

Using git pre-commit hooks, you can run code against your commits (or pushes) automatically.You can add whatever code you want in the form of a plugin For example, you can see the pre-commit linter working to remove white space and trailing ends from my code commit:

You have to store these plugins within a .pre-commit-config.yaml file and source a repository for them. The following is how mine is structured:

 1repos:
 2  - repo: https://github.com/pre-commit/pre-commit-hooks
 3    rev: v3.2.0
 4    hooks:
 5      - id: check-yaml
 6      - id: end-of-file-fixer
 7      - id: trailing-whitespace
 8  - repo: https://github.com/thoughtworks/talisman
 9    rev: v1.22.0
10    hooks:
11      - id: talisman-commit
12        entry: cmd --githook pre-commit

As you might have guessed, Talisman is a pre-commit hook that scans your commits (or pushes) for secrets. Talisman, however, also needs the user to install it using a script within the repository. I followed the documentation on the GitHub, but unfortunately I was met with a few errors. Initially, it was unable to install due to there already being a pre-commit hook file in the .git/hooks folder, so after deleting it the installer ran fine and nothing changed.

After that, I was getting the following error:

After modifying my .pre-commit-config.yaml to look like what was previously posted thanks to this post, I was able to get it to work on the command line. However, GitHub desktop was reporting errors when attempting to commit changes citing the following error:

CMD="/c/GitHub/CapstoneFinal/.git/hooks/bin/talisman ${DEBUG_OPTS} --githook pre-commit

This line errored out because it could not find the talisman file, even though it existed in that path. It took a very long time to figure out what was going on, but eventually I decided to delete the pre-commit LEGACY file that was automatically created when installing pre-commit within the .git/hooks folder. After that, I was able to get Talisman to work.

To test out talisman, I simply created two files that held a fake key and attempted to push it to my repository:

As you can see, this isn't within my CI/CD pipeline, but I do believe this is the best method for preventing secrets from being stolen in the first place. I may attempt to distribute this to my VM's in order to incorporate Ansible in some way as there is no way to automatically install pre-commit hooks safely when pulling a repository as it is disabled by default due to security issues. Regardless, I now have secret-scanning incorporated into my DevSecOps infrastructure.

Nikto Web Server Testing (DAST)

Upon researching tools to use for Dynamic Application Security Testing (DAST), nikto came up quite often for its ease of use. If you don't know what nikto is, this excerpt from the nikto2 website explains it well.

Nikto is an Open Source (GPL) web server scanner which performs comprehensive tests against web servers for multiple items, including over 6700 potentially dangerous files/programs, checks for outdated versions of over 1250 servers, and version specific problems on over 270 servers. It also checks for server configuration items such as the presence of multiple index files, HTTP server options, and will attempt to identify installed web servers and software.

I initially ran into errors trying to get nikto to run locally by using the docker image. Despite being on the same logical docker network , my container for my web application and my container for nikto were unable to communicate with each other.

I found out later with OWASP Zap what my issue was (which I will discuss later), but in the mean time I decided to install nikto locally on the agent and run it against my container which worked flawlessly.

Here is the code I used for installing and running nikto in my pipeline:

1- script: |
2    git clone https://github.com/sullo/nikto
3    cd nikto/program
4    ./nikto.pl -h http://localhost:5000 -o $(Build.ArtifactStagingDirectory)/nikto.json    

As you can see, I also output the results to a json file which I later publish as a pipeline artifact. Upon investigating the results, I saw that nikto actually found another route that I had never known about within my own application under /console!.

I believe this is available because my application within flask is running under DEBUG mode. Still, this was very cool to see!

OWASP ZAP (DAST)

When speaking of DAST, OWASP ZAP is often brought up. It is similar to Nikto in that you scan a web server for vulnerabilities.

I initially had a lot of trouble attempting to get it to run locally, and after a long time of troubleshooting, I realized that even though I had both my web application container and the OWASP ZAP container on the same network, I was still attempting to have them communicate over localhost:X. It was only when I ran a docker inspect command on them did I realize that they were assigned IP addresses in the 172.17.x.x range.

After getting them to communicate properly, I was able to see my results of running the scan against my docker container, which was honestly surprising to me. ZAP actually was able to detect both the directory traversal AND the XSS attack, plus many other things! How cool is that!

After getting this to work, I had to figure out how I was going to get this onto my pipeline. I had two choices, either attempt to scan my staging environment or write a docker-compose.yaml file to set up both my application and OWASP ZAP. I decided to try installing it locally and thankfully I found some code by UKHO, which, after some modifications, it worked, and it was able to publish my results in the NUnit format.

However, this took an incredibly long amount of time for my pipeline, so I moved it to a separate pipeline. I still wanted to include it into my pipeline somehow, and that's when I decided to implement it into my release pipeline.

After hours of attempting to convert the .yaml file into the GUI release pipeline, I could not get it to work due to structural differences between release and normal pipelines. That's when I found a dated OWASP ZAP [marketplace extension](marketplace https://marketplace.visualstudio.com/items?itemName=CSE-DevOps.zap-scanner).

This extension was initially broken, but after modifying some of the parts listed in the documentation, I was able to get it running and implement it as part of my release pipeline! Now, if a certain amount of problems are found within my staging environment, the release pipeline will not continue and the release will be stopped.

My release pipeline now looks like this:

Datadog Gates + Monitors

Finally, I implemented Datadog monitors into my pipeline. To do this, I created a few new monitors and tests within Datadog which tracks my memory and CPU time of my Staging environment which ADO will query. If it is within normal limits (a set threshold), Datadog will return OK. Only if datadog returns OK will the pipeline continue to release.

This is to prevent a release into production when there is an obvious memory leak or high CPU utilization.

It turns out you can not query tests, which is why they failed, but it provides a good demonstration of what happens if the query does fails. As such, I only have the CPU Time and memory usage monitors within there.

Conclusion

I believe I am mostly done with my capstone project. I only have two more weeks before I have to present and I need to give some time to my other classes to prepare for final exams. As such, I have determined that I will not be able to hit the stretch goals of Kubernetes and Infrastructure as Code integration into my project.

I still have some more tools I want to check out which may be incorporated by next week's blog post. The focus however is to actually go about fixing all of these vulnerabilities and issues in a separate branch which I will demonstrate to the class. This is a very important step as it will help me actually learn about what was reported by these tools and how to better prepare and protect applications in the future.

Some tools I will still be looking at:

• Stackhawk
• Clair Docker SAST
• Snyk Software Composition Analysis
• Sonarqube Code Quality and Security Scanner

Concept of security with padlock and keys Illustration by Iconscout Store