我有一个云形成模板的AWS批处理POC有6个资源。
使用的角色是:
但即使使用策略“arn:aws:iam::aws:policy/administratoraccess”,我在rin作业时也会得到“CanNotPullContainerError:错误响应from daemon:get https://*********.dkr.ecr.eu-west-1.amazonaws.com/v2/:net/http:request canceled while while while while while while waiting headers(client.timeout exceed while awaiting headers)”。
质疑者:所有都是FARGATE(计算机环境和作业),而不是EC2
AWSTemplateFormatVersion: '2010-09-09'
Description: Creates a POC AWS Batch environment.
Parameters:
Environment:
Type: String
Description: 'Environment Name'
Default: TEST
Subnets:
Type: List<AWS::EC2::Subnet::Id>
Description: 'List of Subnets to boot into'
ImageName:
Type: String
Description: 'Name and tag of Process Container Image'
Default: 'upload:6.0.0'
Resources:
BatchServiceRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchServiceRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service: 'batch.amazonaws.com'
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchContainerRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchContainerRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: 'Allow'
Principal:
Service:
- 'ecs-tasks.amazonaws.com'
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchJobRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchJobRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service: 'ecs-tasks.amazonaws.com'
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchCompute:
Type: "AWS::Batch::ComputeEnvironment"
Properties:
ComputeEnvironmentName: DemoContentInput
ComputeResources:
MaxvCpus: 256
SecurityGroupIds:
- sg-0b33333333333333
Subnets: !Ref Subnets
Type: FARGATE
ServiceRole: !Ref BatchServiceRole
State: ENABLED
Type: Managed
Queue:
Type: "AWS::Batch::JobQueue"
DependsOn: BatchCompute
Properties:
ComputeEnvironmentOrder:
- ComputeEnvironment: DemoContentInput
Order: 1
Priority: 1
State: "ENABLED"
JobQueueName: DemoContentInput
ContentInputJob:
Type: "AWS::Batch::JobDefinition"
Properties:
Type: Container
ContainerProperties:
Command:
- -v
- process
- new-file
- -o
- s3://contents/{content_id}/{content_id}.mp4
Environment:
- Name: SECRETS
Value: !Join [ ':', [ '{{resolve:secretsmanager:common.secrets:SecretString:aws_access_key_id}}', '{{resolve:secretsmanager:common.secrets:SecretString:aws_secret_access_key}}' ] ]
- Name: APPLICATION
Value: upload
- Name: API_KEY
Value: '{{resolve:secretsmanager:common.secrets:SecretString:fluzo.api_key}}'
- Name: CLIENT
Value: upload-container
- Name: ENVIRONMENT
Value: !Ref Environment
- Name: SETTINGS
Value: !Join [ ':', [ '{{resolve:secretsmanager:common.secrets:SecretString:aws_access_key_id}}', '{{resolve:secretsmanager:common.secrets:SecretString:aws_secret_access_key}}', 'upload-container' ] ]
ExecutionRoleArn: 'arn:aws:iam::**********:role/DemoBatchJobRole'
Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref ImageName ] ]
JobRoleArn: !Ref BatchContainerRole
ResourceRequirements:
- Type: VCPU
Value: 1
- Type: MEMORY
Value: 2048
JobDefinitionName: DemoContentInput
PlatformCapabilities:
- FARGATE
RetryStrategy:
Attempts: 1
Timeout:
AttemptDurationSeconds: 600
在AWS::Batch::JobQueue:ContainerProperties:ExecutionRoLearn中,我对arn进行了编码,因为如果write!ref BatchJobRole,我会得到一个错误。但这不是我问这个问题的目的。
问题是如何在我运行作业时避免“CanNotPullContainerError:后台进程的错误响应:Get https://*********.dkr.ecr.eu-west-1.amazonaws.com/v2/:net/http:request canceled while while waiting for connection(client.timeout exceped while awaiting headers)”。
听起来好像你不能从你的子网内部到达internet。
确保:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-connect-internet-gateway/
CloudFormation Templates for AWS Fargate deployments This is a collection of CloudFormation templates for launching containers in Fargate with a variety of different networking approaches. Fargate is
AWS Cloudformation User Guide The open source version of the AWS CloudFormation User Guide License Summary The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 Int
我想知道使用AWS OpsWorks与AWS Beanstalk和AWS CloudFormation的优缺点是什么? 我感兴趣的是一个可以自动伸缩的系统,它可以处理任意数量的并发web请求(从每分钟1000个请求到1000万rpm),包括一个可以自动伸缩的数据库层。 理想情况下,我希望有效地共享一些硬件资源,而不是为每个应用程序提供单独的实例。在过去,我主要使用EC2实例RDS Cloudtop
批处理 本书展示的几个例子中,ElasticSearch提供了高效的批量索引数据的功能,用户只需按批量索引的格式组织数据即可。同时,ElasticSearch也为获取数据和搜索数据提供了批处理功能。值得一提的是,该功能使用方式与批量索引类似,只需把多个请求组合到一起,每个请求可以独立指定索引及索引类型。接下来了解这些功能。 MultiGetMultiGet操作允许用户通过_mget端点在单个请求命
我有以下工作要处理在一定的时间间隔或特别的基础上。 作业中的步骤如下: 我也想要用户界面,在那里我可以触发一个特别的基础上的工作,而且我应该能够提供参数从用户界面。 我想用Spring batch来完成这个任务,但它更多的是用于读->处理->写之类的工作。这里,在第一步中,我正在生成由第二步读取的数据。我不确定我是否还可以使用Spring batch来实现这个,或者有更好的方法来实现这个。
我在表中总共有8条记录,其中6条在spring批处理调用read时可以使用jpareader。现在我将页面大小和块大小设置为1以进行测试。期望作业运行时,它应该进行6次读取调用,然后它应该逐个处理,逐个写入。但实际上发生的是,它只是调用read 4次(从日志中我可以看到这样读取页面0...1)并处理4个,其中一个由于不匹配写入标准而被过滤掉,然后它只是更新了3个记录,作业标记为成功完成。