SpringWebClient：如何将大字节[]流式传输到文件？

Question

SpringRestTemplate似乎无法将响应直接流式传输到文件，而不将其全部缓冲在内存中。使用较新的Spring 5WebClient实现这一点的正确方法是什么？

WebClient client = WebClient.create("https://example.com");
client.get().uri(".../{name}", name).accept(MediaType.APPLICATION_OCTET_STREAM)
                    ....?

我看到人们在restemplate中找到了一些解决此问题的方法，但我更感兴趣的是使用WebClient以正确的方式解决此问题。

有许多使用restemplate下载二进制数据的示例，但几乎所有示例都将字节[]加载到内存中。

Answer 1

static <R> Mono<R> writeBodyToTempFileAndApply(
        final WebClient.ResponseSpec spec,
        final Function<? super Path, ? extends R> function) {
    return using(
            () -> createTempFile(null, null),
            t -> write(spec.bodyToFlux(DataBuffer.class), t)
                    .thenReturn(function.apply(t)),
            t -> {
                try {
                    deleteIfExists(t);
                } catch (final IOException ioe) {
                    throw new RuntimeException(ioe);
                }
            }
    );
}

java prettyprint-override">static <R> Mono<R> pipeBodyAndApply(
        final WebClient.ResponseSpec spec, final ExecutorService executor,
        final Function<? super ReadableByteChannel, ? extends R> function) {
    return using(
            Pipe::open,
            p -> {
                final Future<Disposable> future = executor.submit(
                        () -> write(spec.bodyToFlux(DataBuffer.class), p.sink())
                                .log()
                                .doFinally(s -> {
                                    try {
                                        p.sink().close();
                                        log.debug("p.sink closed");
                                    } catch (final IOException ioe) {
                                        throw new RuntimeException(ioe);
                                    }
                                })
                                .subscribe(DataBufferUtils.releaseConsumer())
                );
                return just(function.apply(p.source()))
                        .log()
                        .doFinally(s -> {
                            try {
                                final Disposable disposable = future.get();
                                assert disposable.isDisposed();
                            } catch (InterruptedException | ExecutionException e) {
                                e.printStackTrace();
                            }
                        });
            },
            p -> {
                try {
                    p.source().close();
                    log.debug("p.source closed");
                } catch (final IOException ioe) {
                    throw new RuntimeException(ioe);
                }
            }
    );
}

Answer 2

我无法测试以下代码是否有效地在内存中缓冲webClient负载的内容。然而，我认为你应该从这里开始：

public Mono<Void> testWebClientStreaming() throws IOException {
    Flux<DataBuffer> stream = 
            webClient
                    .get().accept(MediaType.APPLICATION_OCTET_STREAM)
                    .retrieve()
            .bodyToFlux(DataBuffer.class);
    Path filePath = Paths.get("filename");
    AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(filePath, WRITE);
    return DataBufferUtils.write(stream, asynchronousFileChannel)
            .doOnNext(DataBufferUtils.releaseConsumer())
            .doAfterTerminate(() -> {
                try {
                    asynchronousFileChannel.close();
                } catch (IOException ignored) { }
            }).then();
}

Answer 3

使用最近稳定的Spring WebFlux（5.2.4.1版本，截至撰写时）：

final WebClient client = WebClient.create("https://example.com");
final Flux<DataBuffer> dataBufferFlux = client.get()
        .accept(MediaType.TEXT_HTML)
        .retrieve()
        .bodyToFlux(DataBuffer.class); // the magic happens here

final Path path = FileSystems.getDefault().getPath("target/example.html");
DataBufferUtils
        .write(dataBufferFlux, path, CREATE_NEW)
        .block(); // only block here if the rest of your code is synchronous

对我来说，不明显的部分是bodyToFlux（DataBuffer.class），因为它目前在一个关于Spring文档流的通用部分中提到，在WebClient部分中没有直接引用它。