当前位置: 首页 > 工具软件 > CodeQL > 使用案例 >

CodeQL数据库构建原理分析

上官培
2023-12-01

CodeQL 是一个帮助开发者自动完成安全检查、帮助安全研究者进行变异分析的分析引擎。它由代码数据库和代码语义分析引擎组成,通过将代码抽象为数据查询表保存到代码数据库中,可以方便地运行代码查询。本文的关注点在于CodeQL是如何生成代码数据库。

这里以 java 作为示例语言进行分析

在配置好CodeQL以后,用户目录下的 codeql-home/codeql 文件夹保存了CodeQL的 CLI 部分,它的目录结构如下,这里省略了部分无关文件

.
├── codeql
├── java
│   ├── codeql-extractor.yml
│   ├── semmlecode.dbscheme
│   ├── semmlecode.dbscheme.stats
│   └── tools
│         ├── autobuild-fat.jar
│         ├── autobuild.cmd
│         ├── autobuild.sh
│         ├── codeql-java-agent.jar
│         ├── compiler-tracing.spec
│         ├── macos
│         ├── pre-finalize.sh
│         ├── semmle-extractor-java.jar
│         └── tracing-config.lua
└──── tools
    ├── codeql.jar
    ├── osx64
    ├── test
    └── tracer

CodeQL的入口文件为 codeql ,这是一个 shell 脚本,主要目的就是为调用 codeql.jar 做准备,包括检查环境和配置环境变量。 codeql.jar 是CodeQL的核心文件,包含了命令行解析、数据库创建和查询引擎相关的代码。

这里以创建数据库的指令为例。创建数据库要经过下面三步

initialize  初始化数据库,用到codeql.jar
build       生成trap文件,用到codeql-java-agent.jar,semmle-extractor-java.jar            
finalize    将trap文件导入数据库,用到pre-finalize.sh,codeql.jar

我们按照这个流程,分成三步进行分析

我们新建一个IDEA工程,将 codeql.jar 导入为依赖库,然后编写如下代码

package cokeBeer;

import com.semmle.cli2.CodeQL;
import java.io.File;

public class RunCreate {
    public static void main(String[] args) {
        //参数部分可以自由配置,只要能正常运行database create的参数即可
        String UserHome=System.getProperty("user.home");
        String language="java";
        String command="mvn clean package";
        String ProjectName="java-sec-code";
        String CodeQLHome=String.join(File.separator,UserHome,"codeql-home");
        String SourceRoot=String.join(File.separator,CodeQLHome,"source","java-source");
        String DatabaseRoot=String.join(File.separator,CodeQLHome,"database","java-database");
        String source=String.join(File.separator,SourceRoot,ProjectName);
        String database=String.join(File.separator,DatabaseRoot,ProjectName);
        String[] QLArgs=new String[]{"database","create","-v","--overwrite","-l",language,"-s",source,"-c",command,database};
        //调用CodeQL的入口方法,可以在这里下断点
        CodeQL.main(QLArgs);
    }
}

这里选择 java-sec-code 这个项目作为测试项目。具体选择的项目内容对分析过程没有影响,编译指令正确即可。

在入口方法处打上断点,开始调试,接下来的方法调用过程如下

com.semmle.cli2.CodeQL#main
com.semmle.cli2.picocli.SubcommandMaker#runMain(java.lang.String[])
com.semmle.cli2.picocli.SubcommandMaker#runMain(java.lang.String[], java.util.function.Function<com.semmle.cli2.picocli.SubcommandCommon,java.lang.Integer>, boolean)
java.util.function.Function#apply
com.semmle.cli2.picocli.SubcommandCommon#call
com.semmle.cli2.database.CreateCommand#executeSubcommand

最后是进入到了 CreataeCommmand 类,这个类处理创建数据库相关的操作,这里简化了部分代码,方法逻辑流程如下

protected void executeSubcommand() throws SubcommandDone {
    // 初始化数据库
        this.runPlumbingInProcess(InitCommand.class, new Object[]{this.initOptions, "--source-root=" + this.sourceRoot, "--allow-missing-source-root=" + this.traceCommandOptions.hasWorkingDir(), "--allow-already-existing", "--", this.initOptions.directory});
    ...
        // 运行编译指令
    this.runPlumbingInProcess(TraceCommandCommand.class, new Object[]{threadsOption(this.threads), ramOption(this.ram), this.tracingOptions, this.traceCommandOptions, this.extractorOptionsOptions, indexTracelessOption, multispec, "--", multispec.directory, commandLine});
    ...
    // finalize
        this.runPlumbingInProcess(FinalizeCommand.class, new Object[]{threadsOption(this.threads), ramOption(this.ram), this.finalizeParams, multispec, "--", multispec.directory});
        }
}

我们进入初始化数据库的代码,调用链如下

com.semmle.cli2.picocli.SubcommandCommon#runPlumbingInProcess
com.semmle.cli2.picocli.PlumbingRunner#run
com.semmle.cli2.database.InitCommand#executeSubcommand
com.semmle.cli2.database.InitCommand#initOneDatabase

最后是进入了 InitCommand 类,这个类负责初始化数据库。 initOneDatabase 的代码简化后如下

private void initOneDatabase(String language, Path databaseDir, long linesOfCode, Optional<String> shaAnalyzed) {
    // 搜索extractor
    Map<String, List<Path>> allExtractors = ((ResolveLanguagesResult)this.callPlumbingInProcess(ResolveLanguagesCommand.class, new Object[]{this.options.extractorOptions})).getExtractorRoots();
    List<Path> found = (List)allExtractors.get(language);
    Path packRoot = (Path)found.get(0);
    // 创建extractor对象
    CodeQLExtractor extractor = new CodeQLExtractor(packRoot);
    DbInfo dbInfo = new DbInfo(this.sourceRoot.toString(), extractor.usesUnicodeNewlines(), extractor.getColumnKind(), language, allExtractors, linesOfCode, (String)shaAnalyzed.orElse((Object)null), CodeQLVersion.currentVersion().version);
    // 创建 skeleton
    DatabaseLayout layout = DatabaseLayout.create(databaseDir, dbInfo);
}

运行完成后,数据库目录下会出现 codeql-database.yml 文件

java-sec-code $ tree -L 1
.
├── codeql-database.yml
└── log

从 initalize 部分返回以后,就进入了 build 部分,这里我们先调试几步,调用链如下

com.semmle.cli2.picocli.SubcommandCommon#runPlumbingInProcess
com.semmle.cli2.picocli.PlumbingRunner#run
com.semmle.cli2.database.TraceCommandCommand#executeSubcommand
com.semmle.cli2.database.DatabaseProcessCommandCommon#executeSubcommand

这个 executeSubcommand 方法很长,我们关注他进行的两个关键操作。

一是读取 compile.spec 文件,创建 Tracer ,对应代码如下

TracerSetup tracerSetup = this.getTracerSetup(this.logger(), databases, scratchFolder, logFolder, extractors);

getTracerSetup 里面又调用了 getTracingSpec

extractor.getTracingSpec().get()

内容如下,这里 getTracingSpec 会去找 extractor 根目录下的 tools/compile.spec 文件并读取

public Optional<Path> getTracingSpec() {
    Path tools = this.extractorRoot.resolve("tools");
    Path platformTools = tools.resolve(CodeQLDist.currentPlatform().name());
    Iterator var3 = Arrays.asList(platformTools.resolve("compiler-tracing.spec"), tools.resolve("compiler-tracing.spec")).iterator();

    Path candidate;
    do {
        if (!var3.hasNext()) {
            return Optional.empty();
        }

        candidate = (Path)var3.next();
    } while(!Files.isRegularFile(candidate, new LinkOption[0]) || !Files.isReadable(candidate));

    return Optional.of(candidate);
}

用于示例的是 java 的 extractor ,我们很容易找到对应的 compile.spec ,内容如下

jvm_prepend_arg -javaagent:${config_dir}/codeql-java-agent.jar=ignore-project,java
jvm_prepend_arg -Xbootclasspath/a:${config_dir}/codeql-java-agent.jar

可见CodeQL会在build前准备好调用 code-java-agent.jar 相关的参数

二是创建进程,运行build指令。

Builder8 p = new Builder8(cmdArgs, LogbackUtils.streamFor(this.logger(), "build-stdout", true), LogbackUtils.streamFor(this.logger(), "build-stderr", true), Env.systemEnv().getenv(), workingDir.toFile());
this.env.addToProcess(p);
List<String> cmdProcessor = new ArrayList();
CommandLine.addCommandProcessor(cmdProcessor, this.env.expander);
p.prependArgs(cmdProcessor);
tracerSetup.enableTracing(p);
StreamAppender streamOutAppender = new StreamAppender(Streams.out());

int result;
try {
        LogbackUtils.addAppender(streamOutAppender);
    result = p.execute();
} finally {
    LogbackUtils.removeAppender(streamOutAppender);
}

经过一番设置,进程运行时的命令行如下

codeql-home/codeql/tools/osx64/preload_tracer mvn clean package

关键环境变量如下

CODEQL_EXTRACTOR_JAVA_ROOT -> codeql-home/codeql/java
CODEQL_SCRATCH_DIR -> codeql-home/database/java-database/java-sec-code/working
CODEQL_EXTRACTOR_JAVA_LOG_DIR -> codeql-home/database/java-database/java-sec-code/log
CODEQL_EXTRACTOR_JAVA_SOURCE_ARCHIVE_DIR -> codeql-home/database/java-database/java-sec-code/src
CODEQL_EXTRACTOR_JAVA_TRAP_DIR -> codeql-home/database/java-database/java-sec-code/trap/java
SEMMLE_JAVA_TOOL_OPTIONS -> '-javaagent:codeql-home/codeql/java/tools/codeql-java-agent.jar=ignore-project,java' '-Xbootclasspath/a:codeql-home/codeql/java/tools/codeql-java-agent.jar'

因为这里调用的 preload_tracer 为二进制文件,所以直接分析它的具体行为较为困难。

但是我们可以推测出, preload_tracer 会监控编译的过程。当需要运行 JVM 时, preload_tracer 会添加准备好的 -javaagent 参数,使得 codeql-java-agent.jar 参与到编译过程中去。

所以我们接下来的任务是分析 codeql-java-agent.jar 的行为

1.3 codeql-java-agent.jar

这一部分需要读者对于 java-agent 技术和 ASM 技术有一定了解

java 源文件文件一般使用 javac 作为编译程序,生成类文件。但是 javac 仅仅是一个封装程序,其实际的编译操作是调用 com.sun.tools.javac 包下的类来完成的。如果使用 java-agent 技术,劫持 com.sun.tools.javac 包下的关键方法,就能自定义编译行为。

我们编写如下代码来调试 codeql-java-agent.jar

package cokeBeer;
import com.sun.tools.javac.main.Main;
import com.sun.tools.javac.util.Context;

public class RunAgent {
    public static void main(String[] args) throws Exception{
        Main main=new Main("");
        String[] arg=new String[]{"Test.java"};
        main.compile(arg,new Context());
        System.out.println("run agent");
    }
}

为了调试 codeql-java-agent.jar ,首先将其作为库文件导入IDEA,然后在运行配置中添加 vmoptions 如下

-javaagent:your-codeql-home/codeql/java/tools/codeql-java-agent.jar=ignore-project,java

同时在运行配置中添加环境变量如下

CODEQL_EXTRACTOR_JAVA_ROOT=your-codeql-home/codeql/java
CODEQL_EXTRACTOR_JAVA_LOG_DIR=your-test-dir/log

再找到入口方法 com.semmle.extractor.java.InterceptingAgent#premain 打上断点,就可以开始调试了

public static void premain(String agentArgs, Instrumentation inst) {
    inst.addTransformer(new InterceptingAgent(agentArgs, new Interceptor[0]));
}

这里我们看到 premain 创建了一个 InterceptingAgent 类型的对象,然后添加为 Transformer

我们先看 InterceptingAgent 的构造方法

public InterceptingAgent(String agentArgs, Interceptor... extraInterceptors) {
    // 略去部分无关代码
    Set<String> args = new LinkedHashSet(Arrays.asList(agentArgs.split(",")));
    Iterator var6 = args.iterator();

    while(var6.hasNext()) {
        String arg = (String)var6.next();
        if (!arg.equals("ignore-project")) {
            if (arg.equals("java")) {
                this.interceptors.add(new JavacMainInterceptor());
                this.interceptors.add(new JavacToolInterceptor());
                this.interceptors.add(new ECJInterceptor());
                this.interceptors.add(new TakariLifecycleJdtInterceptor());
                if (Boolean.parseBoolean(System.getenv("CODEQL_EXTRACTOR_JAVA_JSP"))) {
                    this.interceptors.add(new JasperJdtInterceptor());
                    this.interceptors.add(new JasperJspcInterceptor());
                }
            } else if (arg.equals("kotlin")) {
                this.interceptors.add(new KotlinInterceptor());
            } else {
                warn(1, "Unrecognized agent specification: " + arg);
            }
        }
    }
}

可以看出,根据输入参数的不同,会创建不同类型的 Interceptor ,插入到 this.interceptors 去。这里我们的输入参数为 ignore-project,java ,所以会插入 JavacMainInterceptor 和 JavacToolInteceptor

然后我们看 InterceptingAgent 的 tranform 方法,这个方法会在类加载时被系统主动回调

public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) {
    if (loader == null && !bootstrapLoadableClasses.contains(className)) {
        info(2, "Skipping bootstrap-loaded class " + className);
        return null;
    } else if ((!className.startsWith("java/") || className.equals("java/lang/Shutdown")) && !className.startsWith("javax/") && !className.startsWith("sun/")) {
        if (className.startsWith("com/semmle/extractor/java/interceptors/")) {
            info(2, "Skipping intercept handler class " + className);
            return null;
        } else if (className.startsWith("jdk/internal/reflect/")) {
            info(2, "Skipping reflection class " + className);
            return null;
        } else if (className.startsWith("com/semmle/org/objectweb/asm/")) {
            info(2, "Skipping ASM class " + className);
            return null;
        } else {
            boolean intercept = false;
            Iterator var7 = this.interceptors.iterator();

            while(var7.hasNext()) {
                Interceptor i = (Interceptor)var7.next();
                if (i.interceptType(className)) {
                    intercept = true;
                    break;
                }
            }
           //对于需要拦截的类,接下来使用ASM技术进行分析
            ...
        }
    } else {
        info(2, "Skipping system class " + className);
        return null;
    }
}

可以看到 if-else 判断过滤了 java 的内置类,以及 CodeQL 本身包含的类

然后遍历 this.interceptors ,调用 interceptType 方法进行判断。 interceptType 方法要求输入的类名必须和 interceptor 内置的拦截类名一致

例如 JavacMainInterceptor ,它的内置的拦截类就是 com.sun.tools.javac.main.Main

public boolean interceptType(String binaryTypeName) {
        return binaryTypeName.equals("com/sun/tools/javac/main/Main");
}

当成功匹配以后,就使用 ASM 技术,对这个类进行改造。调用 ASM 的代码如下

if (!intercept) {
    info(2, "Skipping class with no interested interceptor: " + className);
    return null;
} else {
    info(1, "Transforming " + className);

    try {
        ClassReader reader = new ClassReader(classfileBuffer);
        if ((reader.getAccess() & 512) != 0) {
            info(2, "Skipping interface " + className);
            return null;
        } else {
            ClassWriter writer = new ClassWriter(reader, 1);
            reader.accept(new RewriteMethods(writer, className, this.collectMemberSignatures(classfileBuffer)), 0);
            return writer.toByteArray();
        }
    } catch (RuntimeException var9) {
        log("ERROR: Exception while processing " + className + ": " + var9);
        var9.printStackTrace(System.out);
        log("Current class loader: " + loader);
        throw var9;
    }
}

这里是创建了一个 RewriteMethods 类型的对象,继承 ASM 中的 ClassVistor ,来重写类文件。这个 RewriteMethods 主要做两件事情,一是拦截并改造特定类方法,这里需要看 visitMethod 方法,它创建了一个 InterceptMethod 类型的对象

public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) {
        return new InterceptMethod(super.visitMethod(access, name, desc, signature, exceptions), access, name, desc);
}

InterceptMethod 又继承了 ASM 中的 MethodVistor ,它实现了 applyInterceptors 方法,内部会尝试遍历 this.interceptors 保存的 Interceptor ,然后调用他们的 intercept 方法。

private void applyInterceptors(boolean before) {
    InterceptingAgent.info(3, "Considering method " + this.name + this.desc + " in " + RewriteMethods.this.binaryTypeName + ".");
    Iterator var2 = InterceptingAgent.this.interceptors.iterator();

    while(var2.hasNext()) {
        Interceptor i = (Interceptor)var2.next();

        try {
            // 这里调用了下面的applyInterceptor
            this.applyInterceptor(i, before);
        } catch (Throwable var5) {
            InterceptingAgent.log("ERROR: Interceptor of type " + i.getClass() + " caused an exception: " + var5);
            var5.printStackTrace(System.out);
        }
    }

}

private void applyInterceptor(Interceptor i, boolean before) {
    if (i.interceptType(RewriteMethods.this.binaryTypeName)) {
        Interceptor.Interception interception = i.intercept(RewriteMethods.this.binaryTypeName, RewriteMethods.this.classMembers, this.name, this.desc, before);
        if (interception != null) {
            InterceptingAgent.info(1, "Interceptor " + i + " wants to call " + interception + " for " + RewriteMethods.this.binaryTypeName + "." + this.name + this.desc + ".");
            this.instrument(interception);
        } else {
            InterceptingAgent.info(2, "Interceptor " + i + " is not interested in " + RewriteMethods.this.binaryTypeName + "." + this.name + this.desc + ".");
        }
    }
}

这里调试时会调用到 JavacMainInterceptor 的 intercept 方法,里面拦截 com.sun.tools.java.main.Main 类型的两个 compile 方法,这两个方法都是负责编译源文件的方法

public Interceptor.Interception intercept(String binaryTypeName, Set<String> classMembers, String methodName, String methodDescriptor, boolean before) {
    if (before) {
        return null;
    } else {
        if (methodName.equals("compile")) {
            if (methodDescriptor.equals("([Ljava/lang/String;Lcom/sun/tools/javac/util/Context;)Lcom/sun/tools/javac/main/Main$Result;")) {
                return new Interceptor.Interception("com/semmle/extractor/java/interceptors/JavacMainInterceptor", "void javacMainResult(Object,String[])", new Interceptor.CallWith[]{CallWith.STACK_TOP, CallWith.FIRST_ARG});
            }

            if (methodDescriptor.equals("([Ljava/lang/String;Lcom/sun/tools/javac/util/Context;)I")) {
                return new Interceptor.Interception("com/semmle/extractor/java/interceptors/JavacMainInterceptor", "int javacMainInt(int,String[])", new Interceptor.CallWith[]{CallWith.FIRST_ARG});
            }
        }

        return null;
    }
}

然后创建对应的 Interception 类型的对象并返回,从 applyInterceptor 方法中看到返回值会被传递给 instrument 方法,这个方法的向类字节码中写入了一个方法调用 SEMMLE_INTERCEPT$0

private void instrument(Interceptor.Interception interception) {
    Integer idx = (Integer)RewriteMethods.this.applicableInterceptions.get(interception);
    if (idx == null) {
        idx = RewriteMethods.this.applicableInterceptions.size();
        RewriteMethods.this.applicableInterceptions.put(interception, idx);
    }

    Interceptor.CallWith[] var3 = interception.callWith();
    int var4 = var3.length;

    for(int var5 = 0; var5 < var4; ++var5) {
        Interceptor.CallWith cw = var3[var5];
        switch (cw) {
            case ALL_ARGS:
                this.loadArgs();
                break;
            case ALL_ARGS_AS_ARRAY:
                this.loadArgArray();
                break;
            case FIRST_ARG:
                this.loadArg(0);
                break;
            case CLASS:
                this.visitLdcInsn(RewriteMethods.this.binaryTypeName);
                break;
            case METHOD_NAME_AND_DESC:
                this.visitLdcInsn(this.name);
                this.visitLdcInsn(this.desc);
                break;
            case STACK_TOP:
                this.visitInsn(89);
                break;
            case THIS:
                if (!this.isStatic && !this.name.equals("<init>")) {
                    this.visitVarInsn(25, 0);
                } else {
                    this.visitInsn(1);
                }
        }
    }

    Method method = Method.getMethod(interception.methodDecl());
    this.visitMethodInsn(184, RewriteMethods.this.binaryTypeName, "SEMMLE_INTERCEPT$" + idx, method.getDescriptor(), false);
}

RewriteMethods 做的第二件事情是创建一个新方法,这个方法就是上面调用的方法 SEMMLE_INTERCEPT$0

这个一部分对应着它的 visitEnd 方法,里面使用 ASM 技术,构造了这个新方法。

为了直观展示,我们直接获取最终转换好的字节码进行反编译。最终发生变化的部分如下

public Result compile(String[] var1, Context var2) {
    Result var10000 = this.compile(var1, var2, List.nil(), (Iterable)null);
    SEMMLE_INTERCEPT$0(var10000, var1);
    return var10000;
}

private static void SEMMLE_INTERCEPT$0(Object var0, String[] var1) {
    Object var10000 = var0;
    String[] var10001 = var1;

    try {
        JavacMainInterceptor.javacMainResult(var10000, var10001);
    } catch (NoClassDefFoundError var2) {
        System.err.println("ERROR: Exception during invocation of Semmle Java compiler. Perhaps you need to put odasa-agent.jar on the boot classpath?");
        var2.printStackTrace(System.err);
    }
}

可以看到,新的 compile 方法获取原 compile 方法的输入参数和编译返回值,然后交给 javacMainResult 处理

@InterceptionMethod
public static void javacMainResult(Object result, String[] args) {
    info(1, "Intercepted javac Main.compile(String[],Context): " + Arrays.toString(args));
    String resultName = result.toString();
    int javacExitCode = getJavacExitCode(resultName);
    int odasaJavacExitCode = Utils.invokeOdasaJavac(javacExitCode, args);
    if (javacExitCode == 0 && odasaJavacExitCode != 0) {
        throw new Error("Fatal extractor error detected. Attempting to abort build commands.");
    }
}

里面调用 Utils.invoke0dasaJavac ,之后的调用链如下

com.semmle.extractor.java.Utils#invokeOdasaJavac(int, java.lang.String[])
com.semmle.extractor.java.Utils#invokeOdasaJavac(int, java.lang.String[], boolean)
com.semmle.extractor.java.Utils#invokeOdasaJavac(int, java.lang.String[], boolean, java.util.Map<java.lang.String,java.lang.String>)

最后一个 invoke0dasaJavac 方法内部首先配置一系列的环境变量、设置命令行参数,参数内容如下

codeql-home/codeql/java/tools/macos/jdk-extractor-java/bin/java
-Dfile.encoding=UTF-8
-Xmx1024M
-Xms256M
--add-opens
java.base/sun.reflect.annotation=ALL-UNNAMED
-classpath
codeql-home/codeql/java/tools/semmle-extractor-java.jar
com.semmle.extractor.java.JavaExtractor
--jdk-version
-1
--javac-args
@@@/your-test-dir/log/ext/javac.args

然后使用这些参数创建一个程序对象并执行

Builder b = new Builder(cmdLine, System.out, System.err);
b.removeEnvVar("JAVA_TOOL_OPTIONS");
Iterator var38 = addEnv.entrySet().iterator();

while(var38.hasNext()) {
        Map.Entry<String, String> entry = (Map.Entry)var38.next();
        b.putEnvVar((String)entry.getKey(), (String)entry.getValue());
}

exitCode = b.execute();

所以这里就是使用CodeQL内置的 java 命令行程序调用 semmle-extractor-java.jar

有了这些参数,我们可以主动调用 semmle-extractor-java.jar 了

运行 semmle-extractor-java.jar 会解析项目源代码,生成 trap 文件

这里我们将 semmle-extractor-java.jar 作为依赖库添加到IDEA

并编写如下代码来调试 semmle-extractor-java.jar ,其中调用参数来自上面的分析过程

package cokeBeer;

import com.semmle.extractor.java.JavaExtractor;

import java.io.File;

public class RunExtractor {
    public static void main(String[] args) {
        String argPath="@@@/your-test-dir/log/ext/javac.args");
        String[] ExtractorArgs=new String[]{"--jdk-version","-1","--javac-args",argPath};
        JavaExtractor.main(ExtractorArgs);
    }
}

为了调试 semmle-extractor-java.jar ,首先将其作为库文件导入IDEA,然后在运行配置中添加环境变量如下

TRAP_FOLDER=your-test-dir/trap/java
SOURCE_ARCHIVE=your-test-dir/src

在入口方法处打上断点,开始调试。 JavaExtractor#main 首先创建一个 JavaExtractor 类型的对象

public static void main(String[] args) {
    String allArgs = StringUtil.glue(" ", args);
    JavaExtractor extractor = new JavaExtractor(args);
    boolean hasJavacErrors = false;

    try {
        hasJavacErrors = !extractor.runExtractor();
    } catch (Throwable var8) {
        ...
    } finally {
        extractor.close();
    }
}

然后运行 com.semmle.extractor.java.JavaExtractor#runExtractor 方法,里面使用 JavacCompiler 对源文件进行解析,然后利用解析信息生成 trap 文件

boolean runExtractor() {
    // 省略了部分日志相关代码
    // 准备编译环境
    Context context = this.output.getContext();
    JavacFileManager.preRegister(context, this.specialSourcepathHandling);
    Arguments arguments = this.setupJavacOptions(context);
    Options.instance(context).put("ignore.symbol.file", "ignore.symbol.file");
    JavaFileManager jfm = (JavaFileManager)context.get(JavaFileManager.class);
    JavaFileManager bfm = jfm instanceof DelegatingJavaFileManager ? ((DelegatingJavaFileManager)jfm).getBaseFileManager() : jfm;
    JavacFileManager dfm = (JavacFileManager)bfm;
    dfm.handleOptions(arguments.getDeferredFileManagerOptions());
    arguments.validate();
    if (jfm.isSupportedOption(Option.MULTIRELEASE.primaryName) == 1) {
        Target target = Target.instance(context);
        List<String> list = List.of(target.multiReleaseValue());
        jfm.handleOption(Option.MULTIRELEASE.primaryName, list.iterator());
    }

    JavaCompiler compiler = JavaCompiler.instance(context);
    compiler.genEndPos = true;
    Set<JavaFileObject> fileObjects = arguments.getFileObjects(); 
    // 解析源文件
    javac_extend.com.sun.tools.javac.util.List<JCTree.JCCompilationUnit> parsedFiles = compiler.parseFiles(fileObjects);
    compiler.enterTrees(compiler.initModules(parsedFiles));
    Queue<Queue<javac_extend.com.sun.tools.javac.comp.Env<AttrContext>>> groupedTodos = Todo.instance(context).groupByFile();

    int prevErr = 0;

    while(true) {
        while(true) {
            JCTree.JCCompilationUnit cu;
            while(true) {
                Queue todo;
                do {
                    cu = null;
                    Iterator var23 = todo.iterator();

                    while(var23.hasNext()) {
                        javac_extend.com.sun.tools.javac.comp.Env<AttrContext> env = (javac_extend.com.sun.tools.javac.comp.Env)var23.next();
                        if (cu == null) {
                            cu = env.toplevel;
                        } else if (cu != env.toplevel) {
                            throw new CatastrophicError("Not grouped by file: CUs " + cu + " and " + env.toplevel);
                        }
                    }
                } while(cu == null);

                try {
                    Queue<javac_extend.com.sun.tools.javac.comp.Env<AttrContext>> queue = compiler.attribute(todo);
                    String envFlowChecks = System.getenv("CODEQL_EXTRACTOR_JAVA_FLOW_CHECKS");
                    if (envFlowChecks == null || Boolean.valueOf(envFlowChecks)) {
                        compiler.flow(queue);
                    }
                    break;
                } catch (StackOverflowError | Exception var36) {
                    this.logThrowable(cu, var36);
                }
            }

            try {
                CharSequence cachedContent = dfm.getCachedContent(cu.getSourceFile());
                if (cachedContent == null) {
                    try {
                        cachedContent = cu.getSourceFile().getCharContent(false);
                    } catch (IOException var37) {
                        this.logThrowable(cu, var37);
                        continue;
                    }
                }

                String contents = ((CharSequence)cachedContent).toString();
                // 抽取解析信息,创建trap文件
                (new CompilationUnitExtractor(this.output, cu, this.dw)).process(contents);
            } catch (StackOverflowError | Exception var38) {
                this.logThrowable(cu, var38);
            }
            break;
        }
    }
}

我们进入最后生成 trap 文件的方法 com.semmle.extractor.java.CompilationUnitExtractor#process

里面创建了 JavaTrapWriter 类型的对象,然后依次调用各种 Extractor ,抽取信息写入 trap 文件

public void process(String contents) {
    JavaFileObject sourceFile = this.compilationUnit.getSourceFile();
    if (sourceFile.getKind() == Kind.SOURCE) {
        File file = PathTransformer.std().canonicalFile(sourceFile.getName());
        String outputPath = ClassFileLocations.getClassFileLocation(sourceFile.getName()).getOutputPath();
        File outputFile = PathTransformer.std().canonicalFile(outputPath);
        this.output.setCurrentSourceFile(outputFile);
        OdasaOutput.TrapLocker trapLocker = this.output.getTrapLockerForCurrentSourceFile();

        try {
            // 创建writer
            OdasaOutput.JavaTrapWriter writer = trapLocker.getTrapWriter();

            try {
                if (writer != null) {
                    OnDemandExtractor onDemand = new OnDemandExtractor(this.output, writer, this.dw);
                    TreeExtender treeExtender = new TreeExtender(file, contents, this.compilationUnit, this.dw);
                    // 抽取编译单元信息
                    this.extractCompilationUnit(contents, writer, onDemand, treeExtender);
                    Iterator var10 = this.compilationUnit.getTypeDecls().iterator();

                    while(var10.hasNext()) {
                        JCTree aClass = (JCTree)var10.next();
                        if (aClass instanceof JCTree.JCClassDecl) {
                             // 抽取AST信息
                            (new ClassDeclExtractor(writer, treeExtender, onDemand, (JCTree.JCClassDecl)aClass, this.compilationUnit, this.dw)).process();
                        }
                    }

                    treeExtender.writeCommentData(writer);
                    // 抽取类、方法的基本信息以及继承和从属信息
                    onDemand.extract();
                    String rootUri = Env.systemEnv().get("CODEQL_EXTRACTOR_JAVA_JSP_ROOT_URI");
                    String destDir = Env.systemEnv().get("CODEQL_EXTRACTOR_JAVA_JSP_DEST_DIR");
                    if (rootUri != null && destDir != null) {
                        String packge = this.compilationUnit.packge.getQualifiedName().toString();
                        String smapClassName = packge + "/" + FileUtil.basename(outputFile);
                        (new SmapExtractor(outputFile, smapClassName, destDir, rootUri, this.output, writer, this.dw)).extract();
                    }
                }
            } catch (Throwable var16) {
                if (writer != null) {
                    try {
                        writer.close();
                    } catch (Throwable var15) {
                        var16.addSuppressed(var15);
                    }
                }

                throw var16;
            }

            if (writer != null) {
                writer.close();
            }
        } catch (Throwable var17) {
            if (trapLocker != null) {
                try {
                    trapLocker.close();
                } catch (Throwable var14) {
                    var17.addSuppressed(var14);
                }
            }

            throw var17;
        }

        if (trapLocker != null) {
            trapLocker.close();
        }
    }

}

先看 extractCompilationUnit 方法,它向 trap 文件写入包名称信息以及导入信息

private void extractCompilationUnit(String contents, TrapWriter writer, OnDemandExtractor onDemand, TreeExtender treeExtender) {
    this.output.writeCurrentSourceFileToSourceArchive(contents);
    TrapWriter.Label compilationUnitId = treeExtender.writeSourceFile(writer);
    TrapWriter.Label packageId = onDemand.getPackageKey(this.compilationUnit.packge);
    writer.addTuple(JavaTable.CuPackage, new Object[]{compilationUnitId, packageId});
    Iterator var7 = this.compilationUnit.getImports().iterator();

    while(var7.hasNext()) {
        JCTree.JCImport i = (JCTree.JCImport)var7.next();
        classifyImport(treeExtender, writer, onDemand, i);
    }

}

然后是 com.semmle.extractor.java.ClassDeclExtractor#process 方法,它访问整个语法树,向 trap 文件写入表达式和语句信息

public void process() {
        this.log.info("Processing file " + this.compilationUnit.getSourceFile().getName());
        this.visitTree(this.classToExtract);
}

然后是 com.semmle.extractor.java.OnDemandExtractor#extract 方法,其内部会调用

com.semmle.extractor.java.OnDemandExtractor#extractModules
com.semmle.extractor.java.OnDemandExtractor#extractJarInfo

分别抽取模块信息和 jar 包清单信息

然后调用 com.semmle.extractor.java.OnDemandExtractor#extractMembersToCurrentWriter 方法,抽取成员变量和成员方法信息

完成分析以后,之前设置的 trap 目录 your-test-dir/trap/java 下就会出现多个 trap.gz 文件,这里我们简单解压一个来分析一下部分内容

源代码

public static void main(String[] args) {
        System.out.println("hello");
}

生成结果

#[email protected]"callable;{#10007}.main({#10020}){#10009}"
#[email protected]"loc,{#10000},2,24,2,27"
locations_default(#10029,#10000,2,24,2,27)
hasLocation(#10028,#10029)
numlines(#10028,3,3,0)
#10030=*
stmts(#10030,0,#10028,0,#10028)
#10031=*
locations_default(#10031,#10000,2,44,4,5)
hasLocation(#10030,#10031)
numlines(#10030,3,3,0)
#10032=*
exprs(#10032,62,#10009,#10028,-1)
callableEnclosingExpr(#10032,#10028)
#10033=*
locations_default(#10033,#10000,2,19,2,22)
hasLocation(#10032,#10033)
numlines(#10032,1,1,0)
#[email protected]"params;{#10028};0"
params(#10034,#10020,0,#10028,#10034)
paramName(#10034,"args")
#[email protected]"loc,{#10000},2,29,2,41"
locations_default(#10035,#10000,2,29,2,41)
hasLocation(#10034,#10035)
#10036=*
exprs(#10036,63,#10020,#10034,-1)
callableEnclosingExpr(#10036,#10028)
#10037=*
locations_default(#10037,#10000,2,29,2,36)
hasLocation(#10036,#10037)
numlines(#10036,1,1,0)
#10038=*
exprs(#10038,62,#10019,#10036,0)
callableEnclosingExpr(#10038,#10028)
#10039=*
locations_default(#10039,#10000,2,29,2,34)
hasLocation(#10038,#10039)
numlines(#10038,1,1,0)
#10040=*
stmts(#10040,14,#10030,0,#10028)
#10041=*
locations_default(#10041,#10000,3,9,3,36)
hasLocation(#10040,#10041)
numlines(#10040,1,1,0)
#10042=*
exprs(#10042,61,#10009,#10040,0)
callableEnclosingExpr(#10042,#10028)
statementEnclosingExpr(#10042,#10040)
#10043=*
locations_default(#10043,#10000,3,9,3,35)
hasLocation(#10042,#10043)
numlines(#10042,1,1,0)
#10044=*
#[email protected]"class;java.io.PrintStream"
exprs(#10044,60,#10045,#10042,-1)
callableEnclosingExpr(#10044,#10028)
statementEnclosingExpr(#10044,#10040)
#10046=*
locations_default(#10046,#10000,3,9,3,18)
hasLocation(#10044,#10046)
numlines(#10044,1,1,0)
#[email protected]"callable;{#10045}.println({#10019}){#10009}"
callableBinding(#10042,#10047)
#10048=*
exprs(#10048,22,#10019,#10042,0)
callableEnclosingExpr(#10048,#10028)
statementEnclosingExpr(#10048,#10040)
#10049=*
locations_default(#10049,#10000,3,28,3,34)
hasLocation(#10048,#10049)
numlines(#10048,1,1,0)
#10050=*
#[email protected]"class;java.lang.System"
exprs(#10050,62,#10051,#10044,-1)
callableEnclosingExpr(#10050,#10028)
statementEnclosingExpr(#10050,#10040)
#10052=*
locations_default(#10052,#10000,3,9,3,14)
hasLocation(#10050,#10052)
numlines(#10050,1,1,0)
#[email protected]"field;{#10051};out"
variableBinding(#10044,#10053)
namestrings("""hello""","hello",#10048)

从最后面的 #10050=* 开始分析,这里表示刷新标签,无具体含义,但是可以被其他变量绑定为 ID

接下来的 #[email protected]"class;java.lang.System" 表示一个全局 gloablID ,其值为 10051

再下来的 exprs(#10050,62,#10051,#10044,-1) 表示向名为 exprs 的代码表中插入一条记录,具体记录的含义可以在上面工作流程概览部分里面列举到的文件 semmlecode.dbscheme 中找到

#keyset[parent,idx]
exprs(
  unique int id: @expr,
  int kind: int ref,
  int typeid: @type ref,
  int parent: @exprparent ref,
  int idx: int ref
);

对应起来就是 id 为 10050 , kind 为 62 , typeid 为 10051 (也就是上面记录的 java.lang.System 类型), parent 为 10044 , idx 为 -1

经过了上面几步, trap 文件成功地被生成了。接下来就是将 trap 文件导入到代码数据库中。

现在进入最后的finalize部分,调用链如下

com.semmle.cli2.picocli.SubcommandCommon#runPlumbingInProcess
com.semmle.cli2.picocli.PlumbingRunner#run
com.semmle.cli2.database.FinalizeCommand#executeSubcommand
com.semmle.cli2.database.FinalizeCommand#finalizeOne

我们看 finalizeOne 方法的实现,它首先运行 pre-finalize.sh 文件,主要目的是为数据库建立索引。然后调用 doTrapImport 方法,导入 trap 文件

private void finalizeOne(DatabaseLayout dbLayout) throws SubcommandDone {
    Path databaseDir = dbLayout.getDatabasePath();
    if (dbLayout.isFinalized()) {
        throw new UserError("Database " + databaseDir + " is already finalized.");
    } else if (!Files.exists(dbLayout.getSourceArchiveRoot(), new LinkOption[0])) {
        if (this.params.skipEmpty()) {
            this.printWarning(this.emptyDatabaseMessage(databaseDir), new Object[0]);
        } else {
            this.printError(this.emptyDatabaseMessage(databaseDir), new Object[0]);
            throw new SubcommandDone(32);
        }
    } else {
        this.foundOneNonEmpty = true;
        // 执行pre-finalize.sh
        if (!this.params.suppressPreFinalize()) {
            dbLayout.getExtractor().getPreFinalizeScript().ifPresent((script) -> {
                Path workingDir = Paths.get(dbLayout.getSourceLocationPrefix());
                this.printProgress("Running pre-finalize script {} in {}.", new Object[]{script, workingDir});
                int result = this.runPlumbingInProcess(TraceCommandCommand.class, new Object[]{"--working-dir=" + workingDir, "--no-tracing", threadsOption(this.importOptions.getThreads()), ramOption(this.importOptions.getRam()), "--", databaseDir, script});
                if (result != 0) {
                    throw new UserError("Failed to execute pre-finalize script in " + databaseDir + " [exit code: " + result + "].");
                }
            });
        }

        writeSourceLocationPrefixTrap(dbLayout);
        List<Path> trapFolders = Collections.singletonList(dbLayout.getTrapFolder());
        doTrapImport(this, dbLayout, this.importOptions, this.privateImportOptions, trapFolders);
        dbLayout.markAsFinalized();
        if (!this.params.suppressCleanup()) {
            this.runPlumbingInProcess(CleanupDatabaseCommand.class, new Object[]{this.params.cleanupParams, "--", databaseDir});
        }

    }
}

接着看到 doTrapImport 方法,里面先获取数据库的 schmema 文件,然后继续调用 import 指令

static void doTrapImport(SubcommandCommon owner, DatabaseLayout dbLayout, ImportOptions importOptions, PrivateImportOptions privateImportOptions, List<Path> trapPaths) {
    owner.printProgress("Running TRAP import for {}...", new Object[]{dbLayout});
    SimpleTimer timer = new SimpleTimer();
    Path dbscheme = importOptions.getDbscheme();
    if (dbscheme == null) {
        Either<Path, String> detectedDbscheme = dbLayout.getExtractor().getDbscheme();
        if (!detectedDbscheme.isLeft()) {
            throw new UserError((String)detectedDbscheme.getRight());
        }

        dbscheme = (Path)detectedDbscheme.getLeft();
    }

    List<Object> importCommandArgs = new ArrayList(Arrays.asList(importOptions.getRam() != null ? ResolveRamCommand.createHeapSizeOption(importOptions.getRam()) : Collections.EMPTY_LIST, "--dbscheme=" + dbscheme, threadsOption(importOptions.getThreads()), privateImportOptions, "--", dbLayout.getDatasetPath()));
    importCommandArgs.addAll(trapPaths);
    int result;
    if (importOptions.getRam() != null) {
        result = owner.spawnPlumbingAsChildProcess(ImportCommand.class, (RamOptions)null, importCommandArgs.toArray());
    } else {
        result = owner.runPlumbingInProcess(ImportCommand.class, importCommandArgs.toArray());
    }

    if (result != 0) {
        throw new UserError("Dataset import for " + dbLayout.getDatasetPath() + " failed with code " + result + ".");
    } else {
        owner.printProgress("TRAP import complete ({}).", new Object[]{timer});
    }
}

import 指令的调用栈如下

com.semmle.cli2.picocli.SubcommandCommon#runPlumbingInProcess
com.semmle.cli2.picocli.PlumbingRunner#run
com.semmle.cli2.database.FinalizeCommand#executeSubcommand
com.semmle.cli2.ql.dataset.ImportCommand#executeSubcommand

executeSubcommand 的实现如下,构建了一个 TrapImporter 类型对象,然后调用 run 方法

protected void executeSubcommand() throws SubcommandDone {
    if (Files.exists(this.datasetDir, new LinkOption[0]) && !FileUtil8.isEmptyDirectory(this.datasetDir)) {
        if (!Files.isDirectory(this.datasetDir, new LinkOption[]{LinkOption.NOFOLLOW_LINKS})) {
            throw new UserError("Dataset " + this.datasetDir + " exists, but is not a directory.");
        }

        if (!Files.isDirectory((new IMBDiskLayout(this.datasetDir, new Context("default"))).getIdPoolDir(), new LinkOption[0])) {
            throw new UserError("Dataset " + this.datasetDir + " has been finalized and does not support further TRAP import.");
        }

        FileUtil8.strictRecursiveDelete(IMBDiskLayout.getCacheDir(this.datasetDir.resolve("default")));
    }

    AtomicBoolean hasErrors = new AtomicBoolean(false);
    TrapImporter importer = new TrapImporter(new TRAPReaderConfig(this.privateOptions.checkUndefinedLabels(), this.privateOptions.checkUnusedLabels(), this.privateOptions.checkRepeatedLabels(), this.privateOptions.checkRedefinedLabels(), this.privateOptions.checkUseBeforeDefinition(), this.privateOptions.locationInStar(), (error) -> {
        hasErrors.set(true);
        this.printError(error, new Object[0]);
    }), this.datasetDir, "default", new CachingMode(), threadsOptionValue(this.threads));

    try {
        importer.run(Arrays.asList(this.trapPaths), this.dbscheme);
    } catch (Throwable var6) {
        try {
            importer.close();
        } catch (Throwable var5) {
            var6.addSuppressed(var5);
        }

        throw var6;
    }

    importer.close();
    if (this.privateOptions.failOnErrors() && hasErrors.get()) {
        this.printError("Aborting as some errors occured during TRAP import.", new Object[0]);
        throw new SubcommandDone(2);
    }
}

run 方法内部最终实现了导入

public void run(List<Path> trapRoots, Path targetDbscheme) {
    AtomicInteger totalNumTrapFilesCounter = new AtomicInteger(0);
    CancellationToken cancelToken = new CancellationToken();
    CompletableFuture<List<TrapTask<LinkTarget[]>>> tasks = this.scanAndLink(trapRoots, totalNumTrapFilesCounter, this.executor);

    try {
        tasks.thenCompose((taskList) -> {
            return ImportTasksProcessor.importTrap(this.loadDbSchemeBinding(targetDbscheme), this.backend, this.executor, this.trapReaderConfig, new LogProgressTracker(totalNumTrapFilesCounter.get()), taskList, cancelToken);
        }).join();
    } catch (CompletionException var7) {
        logger.error("An exception occurred during TRAP import. The database may be partial.", var7);
        throw Exceptions.asUnchecked(var7.getCause());
    }

    this.copyDbSchemeFile(targetDbscheme);
    logger.info("Finished importing trap files.");
}

从上面的分析过程中可以看出,CodeQL其实不需要 java 项目真的可以成功编译,它只需要分析源码获取语法树即可。那么我们可以考虑跳过编译这一步,直接利用 semmle-java-extractor.jar 生成数据库,这正好可以解决某些场景下,只有反编译出的 java 源代码,但是不能成功编译的问题。

这里还是以 java-sec-code 这个项目为例子,为了模拟无源码的环境,下面我们只使用编译好的 jar 包

下面一共用到三个关键文件夹

  • /Users/cokeBeer/decompiled :反编译出的源代码的位置
  • /Users/cokeBeer/test : semmle-extractor-java 的工作文件夹
  • /Users/coekBeer/nonsource :最终生成数据库的位置

首先用IDEA提供的 fernflower 反编译工具对 java-sec-code-1.0.0.jar 进行反编译

java -jar java-decompiler.jar -dgs=1 java-sec-code-1.0.0.jar <your-dst-dir>

这里我使用的目标文件夹为 /Uesrs/cokeBeer/decompiled

然后在 /Uesrs/cokeBeer/decompiled 下找到反编译好的 java-sec-code-1.0.0.jar ,使用下面指令解压

jar -xvf java-sec-code-1.0.0.jar

然后在解压后的文件里面找到 BOOT-INF/classes 文件夹,这里面保存了反编译好的项目文件

classes $ tree -L 1
.
├── application.properties
├── banner.txt
├── create_db.sql
├── logback-online.xml
├── mapper
├── org
├── static
├── templates
└── url

下面我们先进行initialize环节,新建一个文件夹 nonsource ,使用下面的指令初始化数据库

codeql database init -l java --source-root org /Users/cokeBeer/nonsource

这里的 source-root 参数就设置为上面解压出的源代码文件夹 org

回顾我们之前调试的过程,在完成 codeql-java-agent.jar 的处理以后, test 文件夹里面应该出现一个 log 文件夹,里面保存了调用 semmle-extractor-java.jar 需要的参数 javac.args

log $ tree -L 2
.
├── ext
│   ├── javac.args
│   ├── javac.env
│   ├── javac.orig
│   └── javac.properties
└── javac-errors.log

然后我们可以利用这个文件作为输入,调用 semmle-extractor-java.jar

package cokeBeer;

import com.semmle.extractor.java.JavaExtractor;

import java.io.File;

public class RunExtractor {
    public static void main(String[] args) {
        // 在这里用到了
        String argPath="@@@/Users/cokeBeer/test/log/ext/javac.args");
        String[] ExtractorArgs=new String[]{"--jdk-version","-1","--javac-args",argPath};
        JavaExtractor.main(ExtractorArgs);
    }
}

同时需要配置环境变量,说明生成 trap 文件和 src 相关文件的位置,注意这里先输出到 test 文件夹

TRAP_FOLDER=/Users/cokeBeer/test/trap/java
SOURCE_ARCHIVE=/Users/cokeBeer/test/src

我们看一下 javac.args 的内容

-Xprefer:source
-source
1.8
-target
1.8
-classpath
...
-extdirs
...
-endorseddirs
...
-bootclasspath
...
Test.java

它的最后一行传入了编译的目标,那么我们只需要在这里替换了输入文件,就能正确调用

现在先回到 /Users/cokeBeer/decompiled/BOOT-INF/classes 文件夹,使用下面指令找到所有需要编译的 java 文件

find org -name *.java  > sources.txt

sources.txt 可能是相对目录,可以使用 vscode 批量替换为绝对目录

/Users/cokeBeer/decompiled/BOOT-INF/classes/org/joychou/imageConfig.java
/Users/cokeBeer/decompiled/BOOT-INF/classes/org/joychou/Application.java
/Users/cokeBeer/decompiled/BOOT-INF/classes/org/joychou/util/LoginUtils.java
/Users/cokeBeer/decompiled/BOOT-INF/classes/org/joychou/util/HttpUtils.java
/Users/cokeBeer/decompiled/BOOT-INF/classes/org/joychou/util/WebUtils.java
...

然后将输入替换为 @/Users/cokeBeer/decompiled/BOOT-INF/classes/source.txt

-Xprefer:source
-source
1.8
-target
1.8
-classpath
...
-extdirs
...
-endorseddirs
...
-bootclasspath
...
@/Users/cokeBeer/decompiled/BOOT-INF/classes/source.txt

然后运行之前调试 semmle-extractor-java.jar 时配置好的代码,即可在 /Users/cokeBeer/test/trap/java 文件夹下找到 trap 文件

org $ tree -L 2
.
└── joychou
    ├── Application.java.set
    ├── Application.java.trap.gz
    ├── RMI
    ├── config
    ├── controller
    ├── dao
    ├── filter
    ├── imageConfig.java.set
    ├── imageConfig.java.trap.gz
    ├── mapper
    ├── security
    └── util

现在 trap 文件已经生成完毕,最后就是finalize阶段

先将 /Users/cokeBeer/test 下生成的 trap 和 src 文件夹复制到 /Users/cokeBeer/nonsource

然后运行下面的指令生成数据库

codeql database finalize '/Users/cokeBeer/nosource'

完成以后 /Users/cokeBeer/nonsource 目录结构应该如下

nonsource $ tree -L 1
.
├── codeql-database.yml
├── db-java
├── log
└── src.zip

下面到 vscode 中导入数据库,然后编写一个命令注入的污点分析查询

import java
import semmle.code.java.dataflow.FlowSources
import semmle.code.java.dataflow.TaintTracking

abstract class CommandInjectionSink extends DataFlow::Node {}

private class DefaultCommandInjectionSink extends CommandInjectionSink{
    DefaultCommandInjectionSink(){ 
        exists(ConstructorCall cc |cc.getAnArgument()=this.asExpr()|cc.getCallee().getDeclaringType() instanceof TypeProcessBuilder)
    }
}


class CommandInjectionConfiguration extends TaintTracking::Configuration {
  CommandInjectionConfiguration() { this = "CommandInjection" }

  override predicate isSource(DataFlow::Node source) {
    source instanceof RemoteFlowSource
  }

  override predicate isSink(DataFlow::Node sink) {
    sink instanceof CommandInjectionSink
  }
}

from DataFlow::PathNode source, DataFlow::PathNode sink, CommandInjectionConfiguration conf
where conf.hasFlowPath(source, sink)
select source, sink

成功查询出已知漏洞

commandInjection.ql on nonsource - finished in 0 seconds (1 results) [2022/8/11 18:52:16]

关于这个方案的自动化, waderwu 师傅已经实现了,这里附上他的项目 extractor-java

本文从调试的角度分析了CodeQL数据库构建原理,介绍了CodeQL数据库构建过程中用到的一系列文件和参数的作用和含义。同时也演示了一种绕过编译过程的无源码构建数据库的方案

1.CodeQL构建数据库的实际过程?

使用 java 提供的 API 解析源代码,生成 trap 文件,导入数据库

2.无源代码构建的适用范围

CodeQL在构建 trap 文件时只需要解析源代码,不用真正生成类文件,这部分工作属于编译的前端。所以只需要CodeQL调用的 API 能够成功解析源代码,就能完成无源代码构建。

来源:https://f5.pm/go-122664.html

 类似资料: