Play Framework 2.8的alpakka s3文件下载问题

Question

在S3服务中：

def download(link: String): Source[Option[(Source[ByteString, NotUsed], ObjectMetadata)], NotUsed] = {
    S3.download(TraktrainBucket.DOWNLOAD_BUCKET, link)
}

在控制器中：

          val source = s3Service.download(link).map(s => s.map(_._1))
          val trackName = "track name"
          val filename = trackName.replaceAll("[^A-Za-z0-9 \\-.]", "") + (if (track.drumKit) ".zip" else ".mp3")
          val disposition = "attachment; filename=\"" + filename + "\""
          Result(
            header = ResponseHeader(200, Map("Content-Disposition" -> disposition)),
            body = HttpEntity.Streamed(source.flatMapConcat(_.getOrElse(Source.empty)), None, Some("application/octet-stream"))
          )

另外，我还有一个上传程序（它需要一个mp3文件，用ffmpeg处理它，然后将它上传到s3，就像这样：

def richUploadMp3(extension: String, checkFunction: (String, Option[String]) => Boolean, cannedAcl: CannedAcl, bucket: String) = producerAction(parse.multipartFormData(handleFilePartAsFile)).async { implicit request =>
    val s3Filename = request.user.get.id + "/" + java.util.UUID.randomUUID.toString + "." + extension
    val s3FilenameTagged = request.user.get.id + "/" + java.util.UUID.randomUUID.toString + "." + extension
    val fileOption = request.body.file("file").map {
      case FilePart(key, filename, contentType, file, _, _) =>
        logger.info(s"key = ${key}, filename = ${filename}, contentType = ${contentType}, file = $file")
        if(checkFunction(filename, contentType)) {
          val taggedFile = audioService.putTag(file)
          for {
            mp3 <- FileIO.fromPath(file.toPath).runWith(s3Service.uploadSink(s3Filename, cannedAcl, TraktrainBucket.DOWNLOAD_BUCKET))
            mp3Tagged <- FileIO.fromPath(taggedFile.toPath).runWith(s3Service.uploadSink(s3FilenameTagged, cannedAcl, TraktrainBucket.STREAMING_BUCKET))
          } yield (mp3, mp3Tagged, file, taggedFile)
        } else {
          throw new Exception("Upload failed")
        }
    }
    fileOption match {
      case Some(opt) => opt.map(o => {
        o._3.delete()
        o._4.delete()
        Ok(Json.toJson(Seq(s3Filename, s3FilenameTagged)))
      })
      case _ => Future.successful(BadRequest("ERROR"))
    }
  }

exceeded configured max-open-requests value of 1024

akka {
  loglevel = DEBUG

  http {
    client {
      connecting-timeout = 5 s
      idle-timeout = 5 s
      parsing {
        max-content-length = 3000m
      }
    }
    server {
      parsing {
        max-content-length = 3000m
      }
    }

    host-connection-pool {
      max-open-requests = 1024
      idle-timeout = 10 s
      client {
        connecting-timeout = 10 s
        idle-timeout = 10 s
      }
    }
  }
}

Response stream for [GET /free/642241] failed with 'TCP idle-timeout encountered on connection to [s3.us-west-2.amazonaws.com:443], no bytes passed in the last 10 seconds'. Aborting connection.

val AkkaVersion = "2.5.31"
val AkkaHttpVersion = "10.1.12"
libraryDependencies ++= Seq(
  "com.lightbend.akka" %% "akka-stream-alpakka-s3" % "2.0.1",
  "com.typesafe.akka" %% "akka-stream" % AkkaVersion,
  "com.typesafe.akka" %% "akka-http" % AkkaHttpVersion,
  "com.typesafe.akka" %% "akka-http-xml" % AkkaHttpVersion
)

Answer 1

检查为后端Akka HTTP服务器制作的播放设置中的设置。如果RequestTimeout设置为Infinite，这是默认值，请将其更改为应用程序的合理时间限制。这可能会在连接中出现故障时导致连接长时间运行。如下文所述，这些配置覆盖了Akka配置。

https://www.playframework.com/documentation/2.8.x/settingsakkahttp

注意：Akka HTTP有许多超时配置，您可以使用这些配置来保护应用程序免受攻击或编程错误。正在运行的Akka HTTP服务器将自动识别所有这些Akka配置。例如，如果您有如下的idle-timeout和request-timeout配置:akka.http.server.idle-timeout=20sakka.http.server.request-timeout=30s它们将被自动识别。请记住，上面列出的播放配置将覆盖Akka的配置。

null

为什么这种情况只发生在Akka Http上而不发生在[插入其他客户端]上

默认情况下，许多Java HTTP客户端不会对所使用的某些资源设置限制。例如。有些客户端永远不会将请求排队，而只是在所有池连接当前都忙的情况下打开到服务器的另一个连接。然而，这可能只是将问题从客户端转移到服务器端。此外，使用过多的连接会导致网络性能下降，因为更多的连接会争夺带宽。

还要查看Akka HTTP的S3示例页面

# Defines the default time period within which the application has to
# produce an HttpResponse for any given HttpRequest it received.
# The timeout begins to run when the *end* of the request has been
# received, so even potentially long uploads can have a short timeout.
# Set to `infinite` to completely disable request timeout checking.
#
# Make sure this timeout is smaller than the idle-timeout, otherwise,
# the idle-timeout will kick in first and reset the TCP connection
# without a response.
#
# If this setting is not `infinite` the HTTP server layer attaches a
# `Timeout-Access` header to the request, which enables programmatic
# customization of the timeout period and timeout response for each
# request individually.
request-timeout = 20 s