Friday, November 27, 2015

How docker made security testing a lot easier ?

Let me start by saying, I love docker. If you don't know what it is, quoting Wikipedia:
"Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux.[5]Docker uses resource isolation features of the Linux kernel such as cgroups and kernel namespaces to allow independent "containers" to run within a single Linux instance, avoiding the overhead of starting and maintainingvirtual machines.[6]"

As a pentester, I use docker mainly for 3 stuff:

  • Run hairy applications, like Metasploit and miasm. I've had quite few problems in the past with dependencies and running services. Having a dedicated containers allows to have a clean system to run that single application and which won't get impacted by recurrent changes on your own system
  • Develop new tools, this is the standard use of docker, nothing fancy here. I really recommend coupling with management tools Fabric or Chef.
  • Deploy rapid test environment for security purposes, to illustrate this one, I'll write about my last encounter about an undisclosed command injection in Splunk.
During an internal assessment, several vulnerable Splunk instances were identified suffering from an authenticated command injection. After retrieving valid credentials through using a heartbleed vulnerability from another system, the only way was to find and exploit the command injection.

The initial advisory stated ZDI-14-053:
This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Splunk. User interaction is required to exploit this vulnerability in that the target must visit a malicious page or open a malicious file.
The specific flaw exists within the advanced search functionality. Using a multi-staged attack, it is possible to execute arbitrary commands on the underlying operating system by sending a malformed string to the "runshellscript" script. This vulnerability allows an attacker to execute code under the context of the process.

To identify the vulnerability, I had to have access to a vulnerable version and hopefully have access to a test environment. Weirdly the advisory mentions the need to have user interaction, I don't know if that is a mistake or if I've simply found a way to exploit it without any user interaction.

Hopefully for us, Splunk allowed to retrieve older Splunk version and while searching the Docker Hub, Dockerfile scripts existed for newer version of Splunk.

All we needed to do is change the Dockerfile to run the older splunk version:

FROM ubuntu:trusty

MAINTAINER Denis Gladkikh <>


ENV SPLUNK_HOME /opt/splunk

# add splunk:splunk user
RUN groupadd -r ${SPLUNK_GROUP} \
&& useradd -r -m -g ${SPLUNK_GROUP} ${SPLUNK_USER}

# make the "en_US.UTF-8" locale so splunk will be utf-8 enabled by default
RUN apt-get update && apt-get install -y locales \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8

# pdfgen dependency
RUN apt-get install -y libgssapi-krb5-2

# Download official Splunk release, verify checksum and unzip in /opt/splunk
# Also backup etc folder, so it will be later copied to the linked volume
RUN apt-get install -y wget \
&& mkdir -p ${SPLUNK_HOME} \
&& (cd /tmp && md5sum -c ${SPLUNK_FILENAME}.md5) \
&& tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \
&& rm /tmp/${SPLUNK_FILENAME} \
&& rm /tmp/${SPLUNK_FILENAME}.md5 \
&& apt-get purge -y --auto-remove wget \
&& mkdir -p /var/opt/splunk \
&& rm -fR ${SPLUNK_HOME}/etc \
&& rm -rf /var/lib/apt/lists/*

COPY /sbin/
RUN chmod +x /sbin/

# Ports Splunk Web, Splunk Daemon, KVStore, Splunk Indexing Port, Network Input, HTTP Event Collector
EXPOSE 8000/tcp 8089/tcp 8191/tcp 9997/tcp 1514 8088/tcp

WORKDIR /opt/splunk

# Configurations folder, var folder for everyting (indexes, logs, kvstore)
VOLUME [ "/opt/splunk/etc", "/opt/splunk/var" ]

ENTRYPOINT ["/sbin/"]
CMD ["start-service"]
set -e
if [ "$1" = 'splunk' ]; then
  sudo -HEu ${SPLUNK_USER} ${SPLUNK_HOME}/bin/splunk "$@"
elif [ "$1" = 'start-service' ]; then
  # If user changed SPLUNK_USER to root we want to change permission for SPLUNK_HOME
  if [[ "${SPLUNK_USER}:${SPLUNK_GROUP}" != "$(stat --format %U:%G ${SPLUNK_HOME})" ]]; then
  # If these files are different override etc folder (possible that this is upgrade or first start cases)
  # Also override ownership of these files to splunk:splunk
  if ! $(cmp --silent /var/opt/splunk/etc/splunk.version ${SPLUNK_HOME}/etc/splunk.version); then
    cp -fR /var/opt/splunk/etc ${SPLUNK_HOME}
  sudo -HEu ${SPLUNK_USER} ${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt
  trap "sudo -HEu ${SPLUNK_USER} ${SPLUNK_HOME}/bin/splunk stop" SIGINT SIGTERM EXIT
  if [[ -n ${SPLUNK_FORWARD_SERVER} ]]; then
    if ! sudo -HEu ${SPLUNK_USER} ${SPLUNK_HOME}/bin/splunk list forward-server -auth admin:changeme | grep -q "${SPLUNK_FORWARD_SERVER}"; then
      sudo -HEu ${SPLUNK_USER} ${SPLUNK_HOME}/bin/splunk add forward-server "${SPLUNK_FORWARD_SERVER}" -auth admin:changeme
  sudo -HEu ${SPLUNK_USER} tail -n 0 -f ${SPLUNK_HOME}/var/log/splunk/splunkd_stderr.log &


  image: busybox
    - /opt/splunk/etc
    - /opt/splunk/var
  #image: outcoldman/splunk:6.3.1
  build: .
  hostname: splunk
    - vsplunk
    - 8000:8000

Now we need to start chasing the vulnerability, so we start by looking for the potentially vulnerable files to try to understand their logic:
/splunk/splunk-5.0.4$ find . -name -o -name runshellscript*

And here are the source code of both files:

# simple script that writes parameters 0-7 to $SPLUNK_HOME/bin/scripts/echo_output.txt
read sessionKey
echo "'$0' '$1' '$2' '$3' '$4' '$5' '$6' '$7' '$8' '$sessionKey'" >> "$SPLUNK_HOME/bin/scripts/echo_output.txt"

#   Version 4.0
import os, re, sys, urllib
import splunk.Intersplunk, splunk.mining.dcutils as dcu
import subprocess
from subprocess import PIPE, STDOUT
logger    = dcu.getLogger()
mswindows = (sys.platform == "win32")
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()
# These values will be sent to the shell script:
# $0 = scriptname
# $1 = number of events returned
# $2 = search terms
# $3 = fully qualified query string
# $4 = name of saved splunk
# $5 = trigger reason (i.e. "The number of events was greater than 1")
# $6 = link to saved search
# $7 = DEPRECATED - empty string argument
# $8 = file where the results for this search are stored(contains raw results)
# 5 corresponds to the 6th arg passed to this python script which includes the
# name of this script and the path where the user's script is located
# The script will also receive args via stdin - currently only the session
# key which it can use to communicate back to splunkd is sent via stdin.
# The format for stdin args is as follows:
# <url-encoded-name>=<url-encoded-value>\n
# e.g.
# sessionKey=0729f8e0d4edf7ae18327da6a9976596
# otherArg=123456
# <eof>

if len(sys.argv) < 10:
    splunk.Intersplunk.generateErrorResults("Missing arguments to operator 'runshellscript', expected at least 10, got %i." % len(sys.argv))
script   = sys.argv[1]
if len(script) == 0:
    splunk.Intersplunk.generateErrorResults("Empty string is not a valid script name")
if script[0] == "'" and script[-1] == "'":
    script = script[1:-1]
sharedStorage = settings.get('sharedStorage', splunk.Intersplunk.splunkHome())
if len(sys.argv) > 10:
   path = sys.argv[10]   # the tenth arg is going to be the file
   baseStorage   = os.path.join(sharedStorage, 'var', 'run', 'splunk')
   path          = os.path.join(baseStorage, 'dispatch', sys.argv[9], 'results.csv.gz')

# ensure nothing dangerous
# keep this rule in agreement with etc/system/default/restmap.conf for sane UI
# experience in manager when editing alerts (SPL-49225)
if ".." in script or "/" in script or "\\" in script:
    results = splunk.Intersplunk.generateErrorResults('Script location cannot contain "..", "/", or "\\"')
    # look for scripts first in the app's bin/scripts/ dir, if that fails try SPLUNK_HOME/bin/scripts
    namespace  = settings.get("namespace", None)
    sessionKey = settings.get("sessionKey", None)
    scriptName = script
    if not namespace == None :
        script = os.path.join(sharedStorage,"etc","apps",namespace,"bin","scripts",scriptName)
    # if we fail to find script in SPLUNK_HOME/etc/apps/<app>/bin/scripts - look in SPLUNK_HOME/bin/scripts
    if namespace == None or not os.path.exists(script):
        script = os.path.join(splunk.Intersplunk.splunkHome(),"bin","scripts",scriptName)
    if not os.path.exists(script):
        results = splunk.Intersplunk.generateErrorResults('Cannot find script at ' + script)
        stdin_data = ''
        cmd_args = sys.argv[1:]
        # make sure cmd_args has length of 9
        cmd_args    = cmd_args[:9]
        for i in xrange(9-len(cmd_args)):
        cmd_args[0] = script
        cmd_args[8] = path
        stdin_data = "sessionKey=" + urllib.quote(sessionKey) + "\n"
        # strip any single/double quoting        
        for i in xrange(len(cmd_args)):
            if len(cmd_args[i]) > 2 and ((cmd_args[i][0] == '"' and cmd_args[i][-1] == '"') or (cmd_args[i][0] == "'" and cmd_args[i][-1] == "'")):
                 cmd_args[i] = cmd_args[i][1:-1]
        # python's call(..., shell=True,...)  - is broken so we emulate it ourselves
        shell_cmd   = ["/bin/sh"]
            shell_cmd = [os.environ.get("COMSPEC", "cmd.exe"), "/c"]
        # try to read the interpreter from the first line of the file
            f = open(script)
            line = f.readline().rstrip("\r\n")
            if line.startswith("#!"):
                # Emulate UNIX rules for "#!" lines:
                # 1. Any whitespace (just space and tab, actually) after
                #    the "!" is ignored.  Also whitespace at the end of
                #    the line is dropped
                # 2. Anything up to the next whitespace is the interpreter
                # 3. If there is anything after this whitespace it's
                #    considered to be the argument to pass to the interpreter.
                #    Note that this parsing is very simple -- no quoting
                #    is interpreted and only one argument is parsed.  This
                #    is to match
                line = line[2:].strip(" \t")
                if line != "":
                    arg_loc = line.replace("\t", " ").find(" ")
                    if arg_loc == -1:
                        shell_cmd = [ line ]
                        shell_cmd = [ line[0:arg_loc], line[arg_loc + 1:].lstrip(" \t") ]
        except Exception, e:
        # pass args as env variables too - this is to ensure that args are properly passed in windows
        for i in xrange(len(cmd_args)):
            os.environ['SPLUNK_ARG_' + str(i)] = cmd_args[i]
            p = None
            if mswindows and os.path.split(shell_cmd[0])[1].lower() == "cmd.exe":
                # python seems to get argument escaping for windows wrong too - so we have to fix it ourselves yet again
                # in windows if args to cmd contain spaces the entire sting should be quoted for example:
                # cmd.exe /c " "c:\dir with spaces\" "arg with spaces" "more args with space or special chars" "
                # for more info read cmd.exe /? (the section about quoting and arg processing)
                for i in xrange(0, len(cmd_args)):
                    if not (cmd_args[i].startswith('"') and cmd_args[i].endswith('"')):
                         cmd_args[i] = '"' + cmd_args[i] + '"'
                cmd2run      = shell_cmd[0]+' '+shell_cmd[1]+' " '+ ' '.join(cmd_args) + ' "'
      "runshellscript: " + cmd2run)
                p = subprocess.Popen(cmd2run, stdin=PIPE, stdout=PIPE, stderr=STDOUT, shell=False)
      "runshellscript: " + str(shell_cmd + cmd_args))
                if  mswindows:  # windows doesn't support close_fds param
                    p = subprocess.Popen(shell_cmd + cmd_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT, shell=False)
                    p = subprocess.Popen(shell_cmd + cmd_args, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True, shell=False)
            if p != None:
        except OSError, e:
            results = splunk.Intersplunk.generateErrorResults('Error while executing script ' + str(e))
splunk.Intersplunk.outputResults( results )

After few trials and errors we found a way to exploit the command injection. Basically the vulnerability exists in the script. The script writes it's output to $SPLUNK_HOME/bin/scripts/echo_output.txt file.

The allows however to run all files in the folder $SPLUNK_HOME/bin/scripts, including the

To exploit it, all we need to do is call script to inject valid commands and then call echo_output.txt file: