LUA--AWS分布式代理利用NGINX替换AWS请求头

白昊乾

2023-12-01

1/标准请求头

Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXX/20170522/us-east-1/s3/aws4_request,SignedHeaders=content-md5;
host;
x-amz-content-sha256;
x-amz-date;
x-amz-decoded-content-length,Signatue=xxxxxxxxxxxxx

2/python 封包格式例子

headers = {'Content-Type':content_type,
'X-Amz-Date':amz_date,
'X-Amz-Target':amz_target,
'Authorization':authorization_header}

即整个 auth 认证头要遍历出来，认证参数包含下面

1/包含x amx 时间信息
2/哈希值
3/标志值（需要替换：第一项滴第五支项）

3/代理 AWS 设置（在这里实施替换标志头操作）

nginx.conf

#加入auth 输出 认证头
# set output headers
location /auth{
more_set_headers ’Authorization’:authorization_header
}

location / {
underscores_in_headers on;  // 允许headers fields 里的下划线
# 设置主机头和客户端真实地址 
proxy_set_header        Host $s3_bucket;
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For           
$proxy_add_x_forwarded_for;
add_header              X-Cache-Status 
$upstream_cache_status;
proxy_hide_header       x-amz-id-2;
proxy_hide_header       x-amz-request-id;
proxy_hide_header       Set-Cookie;
proxy_ignore_headers    "Set-Cookie";
proxy_intercept_errors on;
｝

4/auth 认证头字符格式定义

************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************

 authorization_header = algorithm + ' 
 ' + 'Credential=' +  access_key + '/' + 
 credential_scope + ', ' + 
 'SignedHeaders=' +  
 signed_headers + ', 
 ' + 'Signature=' + signature
headers = {'x-amz-date': amzdate, 
'Authorization':  authorization_header}

5/官方PYTHON请求发出和响应处理（请求无需处理由ng代理完成，返回因为替换了参数，需要先读取服务器真实返回然后再向客户端响应）

* SEND THE REQUEST *

print('\nBEGIN    
print('Request URL = ' + endpoint)
r = requests.post(
endpoint,   
data=request_parameters,  headers=headers)
print('Response code: %d\n' % r.status_code)
print(r.text)

6/nginx 清除/替换请求和响应头指定项操作说明

调用接口模块：more_set_headers

7/nginx c 源码层 header 处理

src/ HTTP/ ngx_http_header_filter_module.c文件

8/nginx header 下划线处理

1/nginx.conf 配置
http {
include       mime.types;
default_type  application/octet-stream;
sendfile        on;
underscores_in_headers on;
keepalive_timeout  65;
}

2/源码直接修改
ngx_http_parse_header_line(
ngx_http_request_t *r,  ngx_buf_t *b,
ngx_uint_t allow_underscores)

if (ch == '_') {
if (allow_underscores) {
    hash = ngx_hash(0, ch);
    r->lowcase_header[0] = ch;
    i = 1;
} else {
    r->invalid_header = 1;
 }
    break;
}

9/AUTH 参数生成说明
官方说明

# Step 4: Create the canonical headers and signed headers. Header names
创建主机信息
# must be trimmed and lowercase, and sorted in code point order from
字符需转小写
# low to high. Note that there is a trailing \n.
canonical_headers = 'host:' + host + '\n' + 'x-amz-date:' + amzdate + '\n'
加结束符 '\n'
# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers lists those that you want to be included in the 
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'host;x-amz-date'

# Step 6: Create payload hash (hash of the request body content). For GET
# requests, the payload is an empty string ("").
请求操作时 此项为空
payload_hash = hashlib.sha256(('').encode('utf- 8')).hexdigest()

# Step 7: Combine elements to create canonical request
拼接参数，创建请求
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash


# ************* TASK 2: CREATE THE STRING TO    SIGN*************
匹配 algorithm 字段 并生成哈希值
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()

# ************* TASK 3: CALCULATE THE SIGNATURE *************
将上面参数打包计算出signature
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, datestamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()

注：S4 signature 要用时间截生成，必须取出headers参数重新生成，不能直接替换

* TASK 3: CALCULATE THE SIGNATURE *

Create the signing key using the function defined above.

signing_key = getSignatureKey(secret_key, datestamp, region, service)

Sign the string_to_sign using the signing_key

signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()

10/method lua自动操作，官方定义列表

method = 'POST'
service = 'dynamodb'
host = 'dynamodb.us-west-2.amazonaws.com'
region = 'us-west-2'
endpoint = 'https://dynamodb.us-west-2.amazonaws.com/'

POST requests use a content type header. For DynamoDB,

the content is JSON.

content_type = 'application/x-amz-json-1.0'

DynamoDB requires an x-amz-target header that has this format:

DynamoDB_.

amz_target = 'DynamoDB_20120810.CreateTable'

11/根据官方要求参数，从请求头读出，并调用lua 接口替换auth字段后重新生成 S3 认证头
创建lua文档：new_auth.lua

local awss3auth = require "resty.s3_auth"
local util = require "resty.s3_util"
local test    = require "resty.iresty_test"
local tb = test.new({unit_name="amazon_s3_test"})
local awss3 = require "resty.s3"
local cjson = require "cjson"
-- AWS接入KED ID
-- 改成你要替换的
local AWSAccessKeyId='AKIDEXAMPLE'
local AWSSecretAccessKey="wJalrXUtnFEMI
/K7MDENG+bPxRfiCYEXAMPLEKEY"
local aws4_testsuite_dir = "./test/aws-sig-v4-test-suite"
local debug_file = false
local aws4_debugfile_dir = "./test/aws4_debug"

--重新生成 auth 操作
function tb:init()
self.s3_auth = awss3auth:new(
AWSAccessKeyId,   AWSSecretAccessKey, 

--判断method操作
--初始化数组
local headers = {} 
local function get_method(xamzdate, range)
-- 请NGINX 获取请求头
local headers = ngx.req.get_headers()

-- 遍历头部参数
for k, v in pairs(headers) do
	if type(v) == "table" then
		--输出表格数据
		ngx.say(k, " : ", table.content(v, ","), "</br>")  
	else
		--输出非表格数据
		ngx.say(k, " : ", v, "</br>")
	end
end

local xamzdate = ""
local range = ""
local new_sign = ""
local new_uri = ""
local new_method = ""

--- GET OBJECT
if new_method == "GET" and headers["Range"] == "bytes=0-9"
	headers["x-amz-date"] = xamzdate
	headers["Range"] = range
	local auth, sign = s3_auth:authorization_v4("GET", new_uri, headers, "")
end

--- PUT OBJECT
if new_method == "GET" and headers["x-amz-storage-class"] == "REDUCED_REDUNDANCY"
	headers["x-amz-date"] = "20130524T000000Z"
	headers["Date"] = "Fri, 24 May 2013 00:00:00 GMT"
	headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY"
end

--- GET LIFECYCLE 
if(new_method == "GET" and new_uri == "?lifecycle")
	headers["x-amz-date"] = "20130524T000000Z"
	local auth, sign = s3_auth:authorization_v4("GET", "?lifecycle", headers, "")
end

--- Get Bucket
if new_method == "GET" and new_uri == "?max-keys=2&prefix=J"
	headers["x-amz-date"] = "20130524T000000Z"
	local auth, sign = s3_auth:authorization_v4("GET", "?max-keys=2&prefix=J", headers, "")
end

--头部拼接

end
local function main_fun()
get_method()
end


main_fun()

12/openresty lua nginx.conf 增加 new_auth.lua

LUA--AWS分布式代理利用NGINX替换AWS请求头

* SEND THE REQUEST *

* TASK 3: CALCULATE THE SIGNATURE *

Create the signing key using the function defined above.

Sign the string_to_sign using the signing_key

POST requests use a content type header. For DynamoDB,

the content is JSON.

DynamoDB requires an x-amz-target header that has this format:

DynamoDB_.

相关阅读

相关文章

相关问答

相关文档

LUA--AWS分布式代理利用NGINX替换AWS请求头

************* SEND THE REQUEST *************

************* TASK 3: CALCULATE THE SIGNATURE *************

Create the signing key using the function defined above.

Sign the string_to_sign using the signing_key

POST requests use a content type header. For DynamoDB,

the content is JSON.

DynamoDB requires an x-amz-target header that has this format:

DynamoDB_.

相关阅读

相关文章

相关问答

相关文档

* SEND THE REQUEST *

* TASK 3: CALCULATE THE SIGNATURE *