【Docker】clair镜像扫描的实现

                                                             clair镜像扫描的实现

一、前言

clair扫描的相关基础请先移步我的另外一篇文章镜像安全扫描工具clair与clairctl

这次我们采用clair api方式的扫描，基本思路是

• 打包镜像
• 解压tar包至tomcat的ROOT目录，得到每一个镜像的分层文件及描述文件
• 调用每个镜像的分层描述文件
• 使用clair api按照描述文件中提供的层序提交分层至clair中
• clair依照漏洞-特征数据库进行比对
• 使用clair api并携带最后一层的id，查询漏洞数据
• 删除解压目录

---

二、启动容器

我们需要三个容器，分别是

• Tomcat：给镜像的分层提供下载地址

docker run -d --name tomcat8 -p 6062:8080 -v /opt/deploy/clair/scanImage:/usr/local/tomcat/webapps/ROOT tomcat:jdk8-corretto

• Postgres：存放漏洞-特征匹配数据

docker run -d -e POSTGRES_PASSWORD="password" -p 5432:5432 -v /opt/deploy/clair/postgres/data:/var/lib/postgresql/data --name postgres vmware/postgresql-photon:v1.5.0

• Clair：扫描工具

新建目录/opt/deploy/clair/config,在该目录下新建文件config.yml

  database:
    type: pgsql
    options:
      source: postgresql://postgres:password@127.0.0.1:5432/postgres?sslmode=disable
      cachesize: 16384
  api:
    port: 6060
    healthport: 6061
    timeout: 900s
  updater:
    interval: 2h
  notifier:
    attempts: 3
    renotifyinterval: 2h

docker run --net=host -d -p 6060-6061:6060-6061 -v /opt/deploy/clair/config:/config quay.io/coreos/clair:v2.0.1 -config=/config/config.yml

其中6060为扫描端口，6061为健康检查端口

注意，Postgres必须在clair之前启动，否则clair会连不上数据库，直接停止容器。

clair与postgres正常启动后，clair会自动去拉取漏洞数据插入数据库中，这一过程往往持续几个小时，且由于网络原因，大量漏洞数据无法拉取，此时需要另辟蹊径。

---

三、全部步骤

（1）检查工作

• 判断该镜像所在的仓库是否正常，否则无法拉取该镜像
• 判断tomcat、clair、postgres容器是否正常运行中

（2）调用脚本

    private String executeRemoteShell() {
        String result = null;
        SSH ssh = new SSH(主机ip, 用户名, 密码);
        try {
            boolean isConnect = ssh.connect();
            if (isConnect) {
                result = ssh.executeWithResult(脚本全路径，传入参数);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return result;
    }

#!/bin/bash

baseDir=/opt/deploy/clair

if [ $? -eq 0 ];then
    docker pull ${1} > /dev/null 2>&1
    imageName=`echo ${1} | sed 's!\/!_!g'`  > /dev/null 2>&1
    imageDir="${imageName}"  > /dev/null 2>&1
    mkdir ${baseDir}/scanImage/${imageDir}  > /dev/null 2>&1
    docker save -o ${baseDir}/scanImage/${imageDir}/${imageDir}.tar ${1}  > /dev/null 2>&1
    tar -xvf ${baseDir}/scanImage/${imageDir}/${imageDir}.tar -C ${baseDir}/scanImage/${imageDir}/ > /dev/null 2>&1
    cat ${baseDir}/scanImage/${imageDir}/manifest.json
    exit 0
else
    echo 'scan image failed!'
    exit 6

fi

（3）使用clair api提交镜像分层数据，postLayers为false时，此时已经提交过分层数据，可以利用最后一层的id直接查询漏洞。（postgres会将已经查询过的镜像漏洞存储，提交完分层数据后，可以使用最后一层的id获取漏洞数据）

//将镜像的分层文件提交至clair，进行分析并返回结果
    private ImageScanResult getScanResultFromClair(ImageScan imageScan, boolean postLayers) throws Exception {
        String imageName = imageScan.getImageName();
        String imageTag = imageScan.getImageTag();
        //1.获取manifest层序描述文件内容
        String result = RestTemplateUtils.get(getManifestPath(imageName, imageTag));
        JSONArray jsonArray = JSONArray.parseArray(result);
        if (jsonArray == null || jsonArray.size() == 0) {
            throw new Exception("无法获取或解析镜像层序的描述文件");
        }
        JSONObject jsonObject = (JSONObject) jsonArray.get(0);
        @SuppressWarnings("unchecked")
        List<String> layers = (List<String>) jsonObject.get("Layers");
        //提交的api
        String clairApi = "http://" + systemConfig.getRelayServerIp() + ":" + systemConfig.getClairPort() + "/v1/layers";
        if (postLayers) {
            //上一层id
            String beforePath = "";
            //2.向clair逐层提交
            for (String layer : layers) {
                String name = layer.substring(0, layer.indexOf("/"));
                String path = getAbsoluteLayerPath(layer, imageScan.getImageName(), imageScan.getImageTag());
                String parentName = beforePath;
                String format = "Docker";
                beforePath = name;

                Map<String, Map<String, String>> postParameter = new HashMap<>();
                Map<String, String> layerParameter = new HashMap<>();
                layerParameter.put("Name", name);
                layerParameter.put("Path", path);
                layerParameter.put("ParentName", parentName);
                layerParameter.put("Format", format);
                postParameter.put("Layer", layerParameter);

                JSONObject layerJSONObject = JSONObject.parseObject(JSONObject.toJSONString(postParameter));
                RestTemplateUtils.post(clairApi, layerJSONObject, null);
            }
        }

        //3.所有层提交完毕后，使用最后一层进行查询漏洞
        int last = layers.size() - 1;
        String lastName = layers.get(last).substring(0, layers.get(last).indexOf("/"));
        String leak = RestTemplateUtils.get(clairApi + "/" + lastName + "?features&vulnerabilities");
        JSONObject leakJSONObject = JSONObject.parseObject(leak);
        JSONObject leakLayerJSONObject = leakJSONObject.getJSONObject("Layer");
        JSONArray featuresJSONArray = leakLayerJSONObject.getJSONArray("Features");

        List<Feature> featureList = new ArrayList<>();
        Feature feature;
        for (Object f : featuresJSONArray) {
            JSONObject featureJSONObject = (JSONObject) f;
            feature = new Feature();
            String name = featureJSONObject.getString("Name");
            String namespaceName = featureJSONObject.getString("NamespaceName");
            String versionFormat = featureJSONObject.getString("VersionFormat");
            String version = featureJSONObject.getString("Version");
            String addedBy = featureJSONObject.getString("AddedBy");
            List<Vulnerability> vulnerabilities = new ArrayList<>();
            JSONArray vulnerabilitiesJSONArray = featureJSONObject.getJSONArray("Vulnerabilities");
            if (vulnerabilitiesJSONArray == null || vulnerabilitiesJSONArray.size() == 0) {
                vulnerabilitiesJSONArray = new JSONArray();
            }
            Vulnerability vulnerability;
            for (Object v : vulnerabilitiesJSONArray) {
                JSONObject vulnerabilitiesJSONObject = (JSONObject) v;
                vulnerability = new Vulnerability();
                String vulnerabilityName = vulnerabilitiesJSONObject.getString("Name");
                String vulnerabilityNamespaceName = vulnerabilitiesJSONObject.getString("NamespaceName");
                String vulnerabilityDescription = vulnerabilitiesJSONObject.getString("Description");
                String vulnerabilityLink = vulnerabilitiesJSONObject.getString("Link");
                String vulnerabilitySeverity = vulnerabilitiesJSONObject.getString("Severity");
                String vulnerabilityFixedBy = vulnerabilitiesJSONObject.getString("FixedBy");

                vulnerability.setName(vulnerabilityName);
                vulnerability.setNamespaceName(vulnerabilityNamespaceName);
                vulnerability.setDescription(vulnerabilityDescription);
                vulnerability.setLink(vulnerabilityLink);
                vulnerability.setSeverity(vulnerabilitySeverity);
                vulnerability.setFixedBy(vulnerabilityFixedBy);

                vulnerabilities.add(vulnerability);
            }
            feature.setName(name);
            feature.setNamespaceName(namespaceName);
            feature.setVersionFormat(versionFormat);
            feature.setVersion(version);
            feature.setAddedBy(addedBy);
            feature.setVulnerabilities(vulnerabilities);

            featureList.add(feature);

        }

        //4.封装镜像扫描结果
        ImageScanResult imageScanResult = new ImageScanResult();
        int featureNum = featureList.size();
        long vulnerabilityNum = 0;
        long high = 0;
        long medium = 0;
        long low = 0;
        for (Feature temp : featureList) {
            List<Vulnerability> vulnerabilityList = temp.getVulnerabilities();
            vulnerabilityNum += vulnerabilityList.size();
            high += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("High")).count();
            medium += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Medium")).count();
            low += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Low")).count();
        }
        imageScanResult.setImageName(imageScan.getImageName());
        imageScanResult.setImageTag(imageScan.getImageTag());
        imageScanResult.setFeatureNum(featureNum);
        imageScanResult.setVulnerabilityNum(vulnerabilityNum);
        imageScanResult.setHigh(high);
        imageScanResult.setMedium(medium);
        imageScanResult.setLow(low);
        imageScanResult.setFeatureList(featureList);

        return imageScanResult;

    }

（4）善后工作

清理扫描成功后的文件，这时我们打算清理解压出来的所有的tar包文件，否则解压目录将会异常庞大。博主在清理完一个24G的目录后，仅剩下几M的额外文件。

为什么我们不直接删除所有解压目录？这是因为我们需要留下镜像的分层描述文件（manifest.json），这个文件存放着该镜像的分层信息，包括分层id，分层tar包的地址等。后期我们可以依据此文件拿到最后一层的id，并调用clair api直接获取到该镜像的扫描结果。如果我们将扫描结果单独入库的话，那可以在扫描成功后删除整个解压目录。

    //扫描完成后删除镜像解压目录
    private void delImageDirAfterScan(ImageScan imageScan) {
        String imageIdentification = formatImageNameAndTag(imageScan.getImageName(), imageScan.getImageTag());
        String imageDir = systemConfig.getImageScanShellPath() + "scanImage/" + imageIdentification + "/";

        SSH ssh = new SSH(systemConfig.getRelayServerIp(), systemConfig.getRelayServerUser(), systemConfig.getRelayServerPwd());
        try {
            boolean isConnect = ssh.connect();
            if (isConnect) {
                ssh.execute("find " + imageDir + " -name '*.tar' -type f -print -exec rm -rf {} \\;");
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

    }