http://www.ttlsa.com/nginx/nginx-modules-upload-module/
nginx上传模块
一. nginx upload module原理
官方文档: http://www.grid.net.ru/nginx/upload.en.html
Nginx upload module通过nginx服务来接受用户上传的文件,自动解析请求体中存储的所有文件上传到upload_store指定的目录下。这些文件信息从原始请求体中分离并根据nginx.conf中的配置重新组装好上传参数,交由upload_pass指定的段处理,从而允许处理任意上传文件。每个上传文件中的file字段值被一系列的upload_set_form_field指令值替换。每个上传文件的内容可以从$upload_tmp_path变量读取,或者可以将文件转移到目的目录下。上传的文件移除可以通过upload_cleanup指令控制。如果请求的方法不是POST,模块将返回405错误(405 Not Allowed),该错误提示可以通过error_page指令处理。
具体的过程如下:
用户访问能够选择上传文件的页面
用户提交表单
浏览器把文件和有关文件的信息作为请求的一部分发送给服务器
服务器把文件保存到临时存储目录下upload_store
upload_pass指定的处理表单提交的php页面将文件从upload_store拷贝到持久存储位置
二.nginx upload module配置参数
upload_pass 指明后续处理的php地址。文件中的字段将被分离和取代,包含必要的信息处理上传文件。
upload_resumable 是否启动可恢复上传。
upload_store 指定上传文件存放地址(目录)。目录可以散列,在这种情况下,在nginx启动前,所有的子目录必须存在。
upload_state_store 指定保存上传文件可恢复上传的文件状态信息目录。目录可以散列,在这种情况下,在nginx启动前,所有的子目录必须存在。
upload_store_access 上传文件的访问权限,user:r是指用户可读
upload_pass_form_field 从表单原样转到后端的参数,可以正则表达式表示。:
$upload_field_name – 原始文件中的字段的名称
upload_pass_form_field “^submit |description ”;
意思是把submit,description这两个字段也原样通过upload_pass传递到后端php处理。如果希望把所有的表单字段都传给后端可以用upload_pass_form_field “^.*$”;
upload_set_form_field 名称和值都可能包含以下特殊变量:
$upload_field_name 表单的name值
$upload_content_type 上传文件的类型
$upload_file_name 客户端上传的原始文件名称
$upload_tmp_path 文件上传后保存在服务端的位置
upload_aggregate_form_field 可以多使用的几个变量,文件接收完毕后生成的并传递到后端
$upload_file_md5 文件的MD5校验值
$upload_file_md5_uc 大写字母表示的MD5校验值
$upload_file_sha1 文件的SHA1校验值
$upload_file_sha1_uc 大写字母表示的SHA1校验值
$upload_file_crc32 16进制表示的文件CRC32值
$upload_file_size 文件大小
$upload_file_number 请求体中的文件序号
这些字段值是在文件成功上传后计算的。
upload_cleanup 如果出现400 404 499 500-505之类的错误,则删除上传的文件
upload_buffer_size 上传缓冲区大小
upload_max_part_header_len 指定头部分最大长度字节。
upload_max_file_size 指定上传文件最大大小,软限制。client_max_body_size硬限制。
upload_limit_rate 上传限速,如果设置为0则表示不限制。
upload_max_output_body_len 超过这个大小,将报403错(Request entity too large)。
upload_tame_arrays 指定文件字段名的方括号是否删除
upload_pass_args 是否转发参数。
三. nginx配置
# wget http://www.nginx.org/download/nginx-1.2.2.tar.gz
# wget http://www.grid.net.ru/nginx/download/nginx_upload_module-2.2.0.tar.gz
# tar zxvf nginx_upload_module-2.2.0.tar.gz -c ../software/
# tar zxvf nginx_upload_module-2.2.0.tar.gz -C ../software/
# ./configure --prefix=/usr/local/nginx --add-module=../nginx_upload_module-2.2.0 --with-http_secure_link_module
# make
# make install
# vi nginx.conf
user www-data;
worker_processes 20;
error_log logs/error.log notice;
working_directory /usr/local/nginx;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
root /www/web/upload;
server {
listen 80;
server_name 192.168.41.129;
error_page 405 =200 @405; //处理405错误
location / {
index index.html index.htm index.php;
}
location @405
{
root /www/web/upload;
}
location ~ \.php$ {
try_files $uri /404.html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
client_max_body_size 100m;
# 上传页面提交到这个location
location /upload {
# 文件上传以后转交给后端的php代码处理
upload_pass @test;
# 上传文件的临时存储位置,目录是散列的,应该存在子目录0 1 2 3 4 5 6 7 8 9
upload_store /www/web/upload/tmp 1;
upload_store_access user:r;
# 设置请求体的字段
upload_set_form_field "${upload_field_name}_name" $upload_file_name;
upload_set_form_field "${upload_field_name}_content_type" $upload_content_type;
upload_set_form_field "${upload_field_name}_path" $upload_tmp_path;
# 指示后端关于上传文件的md5值和文件大小
upload_aggregate_form_field "${upload_field_name}_md5" $upload_file_md5;
upload_aggregate_form_field "${upload_field_name}_size" $upload_file_size;
# 指示原样转到后端的参数,可以用正则表达式表示
upload_pass_form_field "^submit$|^description$";
upload_pass_args on;
}
# 将请求转到后端的地址处理
location @test {
rewrite ^(.*)$ /test.php last;
}
}
}
四. 上传界面
# cat /www/web/upload/upload.html
<html>
<head>
<title>Test upload</title>
</head>
<body>
<h2>Select files to upload</h2>
<form enctype="multipart/form-data" action="/upload" method="post">
<input type="file" name="file1"><br>
<input type="file" name="file2"><br>
<input type="file" name="file3"><br>
<input type="file" name="file4"><br>
<input type="file" name="file5"><br>
<input type="file" name="file6"><br>
<input type="submit" name="submit" value="Upload">
<input type="hidden" name="test" value="value">
</form>
</body>
</html>
五. upload_pass处理内容
# cat test.php //这里只是简单的打印出来,便于先理解上传原理。请对着输出内容理解下nginx upload module配置参数。
<?php
print_r($_POST);
?>
对上传文件的处理请参考:http://cn.php.net/manual/en/features.file-upload.php
六. 测试
http://192.168.41.129/upload.html
输出内容如下所示:
Array
(
[file1_name] => Learning Perl, Sixth Edition.pdf
[file1_content_type] => application/pdf
[file1_path] => /www/web/upload/tmp/4/0000000014
[file1_md5] => 87032cc58109f5c6bb866d2684f9b48c
[file1_size] => 8927511
[file2_name] => Programming Perl, 4th Edition.pdf
[file2_content_type] => application/pdf
[file2_path] => /www/web/upload/tmp/5/0000000015
[file2_md5] => 82a52df177a8912c06af276581cfd5e4
[file2_size] => 21146356
[submit] => Upload
)
注意:需要修改php.ini以下参数
file_uploads on 是否允许通过http上传
upload_max_filesize 8m 允许上传文件的最大大小
post_max_size 8m 通过表单POST给php所能接收的最大值
另外nginx.conf中设置上传文件大小
upload_max_file_size 软限制
client_max_body_size 硬限制
测试过程及结果:
nginx-1.10.3
nginx-upload-module-2.2 (解决了没有to_write的问题)
https://github.com/vkholodkov/nginx-upload-module/tree/2.2
编译:
[root@localhost nginx-1.10.3]# ./configure --prefix=/usr/local/nginx-test-post --add-module=../nginx-upload-module-2.2 --with-http_secure_link_module
测试:
#include <stdio.h>
#include <string.h>
#include <sys/time.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLM *multi_handle;
int still_running;
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
static const char buf[] = "Expect:";
/* Fill in the file upload field. This makes libcurl load data from
the given file name when curl_easy_perform() is called. */
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME, "sendfile",
CURLFORM_FILE, "readme.txt",
CURLFORM_END);
/* Fill in the filename field */
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME, "filename",
CURLFORM_COPYCONTENTS, "readme.txt",
CURLFORM_END);
/* Fill in the submit field too, even if this is rarely needed */
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME, "submit",
CURLFORM_COPYCONTENTS, "send",
CURLFORM_END);
curl = curl_easy_init();
multi_handle = curl_multi_init();
/* initialize custom header list (stating that Expect: 100-continue is not
wanted */
headerlist = curl_slist_append(headerlist, buf);
if(curl && multi_handle) {
/* what URL that receives this POST */
//curl_easy_setopt(curl, CURLOPT_URL, "http://www.test_post.com:8081");
curl_easy_setopt(curl, CURLOPT_URL, "http://192.168.41.129/upload.html");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_multi_add_handle(multi_handle, curl);
curl_multi_perform(multi_handle, &still_running);
do {
struct timeval timeout;
int rc; /* select() return code */
CURLMcode mc; /* curl_multi_fdset() return code */
fd_set fdread;
fd_set fdwrite;
fd_set fdexcep;
int maxfd = -1;
long curl_timeo = -1;
FD_ZERO(&fdread);
FD_ZERO(&fdwrite);
FD_ZERO(&fdexcep);
/* set a suitable timeout to play around with */
timeout.tv_sec = 1;
timeout.tv_usec = 0;
curl_multi_timeout(multi_handle, &curl_timeo);
if(curl_timeo >= 0) {
timeout.tv_sec = curl_timeo / 1000;
if(timeout.tv_sec > 1)
timeout.tv_sec = 1;
else
timeout.tv_usec = (curl_timeo % 1000) * 1000;
}
/* get file descriptors from the transfers */
mc = curl_multi_fdset(multi_handle, &fdread, &fdwrite, &fdexcep, &maxfd);
if(mc != CURLM_OK) {
fprintf(stderr, "curl_multi_fdset() failed, code %d.\n", mc);
break;
}
/* On success the value of maxfd is guaranteed to be >= -1. We call
select(maxfd + 1, ...); specially in case of (maxfd == -1) there are
no fds ready yet so we call select(0, ...) --or Sleep() on Windows--
to sleep 100ms, which is the minimum suggested value in the
curl_multi_fdset() doc. */
if(maxfd == -1) {
#ifdef _WIN32
Sleep(100);
rc = 0;
#else
/* Portable sleep for platforms other than Windows. */
struct timeval wait = { 0, 100 * 1000 }; /* 100ms */
rc = select(0, NULL, NULL, NULL, &wait);
#endif
}
else {
/* Note that on some platforms 'timeout' may be modified by select().
If you need access to the original value save a copy beforehand. */
rc = select(maxfd+1, &fdread, &fdwrite, &fdexcep, &timeout);
}
switch(rc) {
case -1:
/* select error */
break;
case 0:
default:
/* timeout or readable/writable sockets */
printf("perform!\n");
curl_multi_perform(multi_handle, &still_running);
printf("running: %d!\n", still_running);
break;
}
} while(still_running);
curl_multi_cleanup(multi_handle);
/* always cleanup */
curl_easy_cleanup(curl);
/* then cleanup the formpost chain */
curl_formfree(formpost);
/* free slist */
curl_slist_free_all(headerlist);
}
return 0;
}
failed to create output file "/www/web/upload/tmp/1/0000000001" for "readme.txt" (13: Permission denied)
worker 没有权限创建/tmp/0 /tmp/1 /tmp/2 ... /tmp/9目录
chmod 777 tmp -R
输出:
* About to connect() to 172.17.2.101 port 80 (#0)
* Trying 172.17.2.101...
perform!
* Connected to 172.17.2.101 (172.17.2.101) port 80 (#0)
> POST /upload.html HTTP/1.1
Host: 172.17.2.101
Accept: */*
Content-Length: 430
Content-Type: multipart/form-data; boundary=----------------------------320a0dcae287
running: 1!
perform!
running: 1!
perform!
running: 1!
perform!
running: 1!
perform!
< HTTP/1.1 404 Not Found
< Server: nginx/1.10.3
< Date: Thu, 13 Jul 2017 08:26:01 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
* HTTP error before end of send, stop sending
<
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.10.3</center>
</body>
</html>
* Closing connection 0
running: 0!