MongoDB GridFS 数据读取效率 benchmark
詹钊
GridFS 是 MongoDB 下的一个子模块，利用这一模块，可以实现使用MongoDB存储文件，MongoDB 二进制包的bin目录下的mongofile 命令即可模拟实现该功能。
原文链接：http://lgone.com/html/y2010/815.html
数据来源：
http://www.coffeepowered.net/2010/02/17/serving-files-out-of-gridfs/
http://www.coffeepowered.net/2010/02/24/serving-files-out-of-gridfs-part-2/
另外一份benchmark数据：http://www.ypass.net/solaris/nginx-gridfs-benchmarks/rawresults.php
目前使用GridFS有以下三种方式：
- 使用脚本读取，这时文件数据会全部读取到脚本中，再调用脚本客户端的输出功能输出给页面。毫无疑问，这种使用方式是效率非常低下的。
- 使用gridfs-fuse使GridFS中存的数据可以通过标准磁盘IO方式进行访问，可以说是真正实现了一个文件系统。
- 使用nginx-gridfs利用Nginx直接读取gridfs中的数据进行发送，这个类似于Nginx的sendfile机制。
文章作者就这几种方式做了测试，得出以下的benchmark数据。
1.1 通过Apache直接读取普通文件
[chris@polaris conf]# ab -n 50000 -c 10 http://advice/images/embed/alliance-60.png
Server Software: Apache/2.2.13
Server Hostname: advice
Server Port: 80
Document Path: /images/embed/normal_alliance-60.png
Document Length: 31596 bytes
Concurrency Level: 10
Time taken for tests: 1.904 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 159463760 bytes
HTML transferred: 158043192 bytes
Requests per second: 2625.37 [#/sec] (mean)
Time per request: 3.809 [ms] (mean)
Time per request: 0.381 [ms] (mean, across all concurrent requests)
Transfer rate: 81767.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.4 1 4
Processing: 1 3 0.5 3 6
Waiting: 0 1 0.4 1 4
Total: 2 4 0.4 4 8
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 4
95% 4
98% 5
99% 5
100% 8 (longest request)
结果：每秒2625次的请求处理，81.77M的吞吐率。
1.2 通过Nginx直接读取普通文件
[chris@polaris conf]# ab -n 50000 -c 10 http://advice:81/images/embed/normal_alliance-60.png
Server Software: nginx/0.8.33
Server Hostname: advice
Server Port: 81
Document Path: /images/embed/normal_alliance-60.png
Document Length: 31596 bytes
Concurrency Level: 10
Time taken for tests: 7.623 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Total transferred: 1590513618 bytes
HTML transferred: 1579863192 bytes
Requests per second: 6559.31 [#/sec] (mean)
Time per request: 1.525 [ms] (mean)
Time per request: 0.152 [ms] (mean, across all concurrent requests)
Transfer rate: 203763.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 9
Processing: 1 1 0.4 1 11
Waiting: 0 0 0.1 0 9
Total: 1 1 0.5 1 12
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 2
90% 2
95% 2
98% 3
99% 3
100% 12 (longest request)
结果：每秒6559次的请求处理，203.763M的吞吐率。
2.通过gridfs-fuse读取gridfs中的数据
[chris@polaris gridfs-fuse]# ab -n 5000 -c 25 http://advice:81/images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Server Software: nginx/0.8.33
Server Hostname: advice
Server Port: 81
Document Path: /images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Document Length: 14332 bytes
Concurrency Level: 25
Time taken for tests: 5.029 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 72725000 bytes
HTML transferred: 71660000 bytes
Requests per second: 994.22 [#/sec] (mean)
Time per request: 25.145 [ms] (mean)
Time per request: 1.006 [ms] (mean, across all concurrent requests)
Transfer rate: 14121.93 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 16 25 1.4 25 52
Waiting: 2 24 1.4 24 52
Total: 17 25 1.4 25 53
Percentage of the requests served within a certain time (ms)
50% 25
66% 25
75% 25
80% 25
90% 25
95% 26
98% 27
99% 32
100% 53 (longest request)
结果：每秒994次的请求处理，14.121M的吞吐率。
3 通过nginx-gridfs模块读取gridfs中的数据
[chris@polaris conf]# ab -n 5000 -c 10 http://advice:81/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Server Software: nginx/0.8.33
Server Hostname: advice
Server Port: 81
Document Path: /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length: 31596 bytes
Concurrency Level: 10
Time taken for tests: 4.613 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 158580000 bytes
HTML transferred: 157980000 bytes
Requests per second: 1083.88 [#/sec] (mean)
Time per request: 9.226 [ms] (mean)
Time per request: 0.923 [ms] (mean, across all concurrent requests)
Transfer rate: 33570.65 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 1 9 4.7 9 103
Waiting: 1 9 4.7 9 102
Total: 2 9 4.7 9 103
Percentage of the requests served within a certain time (ms)
50% 9
66% 9
75% 9
80% 9
90% 9
95% 9
98% 9
99% 11
100% 103 (longest request)
结果：每秒1083次的请求处理，33.57M吞吐率。
4 通过Rails客户端读取数据再发送
[chris@polaris nginx-gridfs]$ ab -n 250 -c 4 http://advice/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Server Software: Apache/2.2.13
Server Hostname: advice
Server Port: 80
Document Path: /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length: 31596 bytes
Concurrency Level: 4
Time taken for tests: 4.646 seconds
Complete requests: 250
Failed requests: 0
Write errors: 0
Total transferred: 7960000 bytes
HTML transferred: 7899000 bytes
Requests per second: 53.81 [#/sec] (mean)
Time per request: 74.338 [ms] (mean)
Time per request: 18.585 [ms] (mean, across all concurrent requests)
Transfer rate: 1673.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 15 74 75.6 34 287
Waiting: 0 72 75.8 30 276
Total: 15 74 75.6 34 288
Percentage of the requests served within a certain time (ms)
50% 34
66% 39
75% 139
80% 192
90% 201
95% 210
98% 239
99% 245
100% 288 (longest request)
结果：每秒53的请求处理，1.673M的吞吐率（效率确实不可能高）
结论：
| Solution | Requests/second | % Apache FS | % Nginx FS | %gridfs-fuse | % Nginx GridFS | % Apache Ruby |
|---|---|---|---|---|---|---|
| Filesystem via Apache | 2625.37 | - | 40.03% | 264.06% | 242.22% | 4,878.96% |
| Filesystem via Nginx | 6559.31 | 249.84% | - | 657.74% | 605.17% | 12,189.76% |
| nginx via gridfs-fuse | 994.22 | 37.86% | 15.15% | - | 91.74% | 1847.65% |
| GridFS via nginx module | 1083.88 | 41.28% | 16.52% | 109.02% | - | 2014.27% |
| Rails metal handler via Passenger | 53.81 | 2.05% | 0.82% | 5.41% | 4.96% | - |
转载：http://blog.nosqlfan.com/html/730.html
类似资料：