小米8融合位置服务器,小米融合云组崔建伟 - 小米融合云实践

1.

2.融合云介绍 • – – – • –

3.融合云介绍 • – – – – Web console • • – HBase • – HBase HDFS • – Http /

4.融合云介绍 Web Console SDK Docker … HBase/HDFS/Kudu/TensorFlow/Spark/Docker…

5.团队管理 (Cloud-Manager) • Organization(Org) – • Team – – Org – Admin/Member – Org admin Team • – CAS – • • • –Org( ): notebook Admin Team Dev_Team Report Team Admin Admin R Tables Bucket/ Objects queues Web console SDK CAS Cloud-Manageruser:teamListresource:org Service

6.团队管理 c c

7.⼩⽶融合云 / (SDS) Snapshot Export (FDS) MR/ Spark (EMQ) (EMQ) SDS Stream

8.结构化存储(SDS) Structure Data Store(SDS) • • HBase • – EntityGroup/Primary Key – • SDK • ACL • – – – Stream –

9.结构化存储(SDS) SDS • SDK – Java/php/python/go/c++/node.js SDK Service Thrift Processor Thrift Processor HttpClient (TTransport) HttpServer (TTransport) flush/HttpRequest read/HttpResponse Multi-platform SDK FrontEnd SDS Object Service Mapping • SDS Service – ACL Quota • Cloud-Manager – • UserTable => TableId => HBase Table SDS Core HBase

10.结构化存储(SDS) SDS • SDS Core – SDS Request HBase Request • Object Mapping – Java Object SDS Request • ACL Multi-platform SDK – FrontEnd – EntityGroup • Quota SDS Object Service Mapping – –ACL Quota SDS Core – Soft Limit • quota HBase •

11.结构化存储(SDS) SDS Core • Entity Group(EG) –: – Entity Group Keys – EG region – EG – EG • • Schema – EntityGroup/Primary Index/Secondary Index/Column • – Bool/Int8/Int16/Float/Double/String/Binary, Set – OrderedBinary •

12.结构化存储(SDS) • – Family/ Family EG Row – HBase coprocessor 01 main_row Data Family • • rowkey • // … 01 Index_row – Eager/Lazy/Immutable Index • Lazy invalid index – Eager/Immutable Index • – HBase table – Themis HBase •https://github.com/xiaomi/themis• :https://github.com/xiaomi/chronos• Index Family

13.结构化存储(SDS) Object Mapping • Java annotation –toRecord:Object => Map–fromRecord:Map=> Object

14.SDS Object Mapping⽰例 Table Schema (SQL)

15.SDS Object Mapping⽰例 Table Schema

16.SDS Object Mapping⽰例

17.SDS Object Mapping⽰例

18.结构化存储(SDS) Stream • – OLAP – F1 Data F2 Image MR • – Stream Family snapshots Snapshoter Snapshot just before Ts • image • hbase coprocessor build images – image • Ts Recovery

19.Web console⽰例

20.结构化存储(SDS) • SDS rest server – 24 Intel(R) Xeon(R) E5620 @2.4 GHz, 64G, • HBase region server – Cpu/ / 8 * 800G ssd 5 • YCSB clientPut Througput/Latency Througput Avg(ms) P95(ms) 0.1k 33738 0.52 7 0.5k 30047 0.46 6 1k 25652 0.46 6 P99(ms) 9 11 10 Get Througput/Latency Througput Avg(ms) 0.1k 30213 0.26 0.5k 27457 0.28 1k 26932 0.29 P95(ms) 4 3 3 P99(ms) 7 9 12

21.对象存储(FDS) File Data Store(FDS) • KB TB • HBase & HDFS • Bucket & Object • Restful API • SDK • • • Quota • 30+ CDN FrontEnd Meta Manager Blob Store HBase HDFS

22.对象存储(FDS) • Bucket Table – Bucket /ACL/Quota • Object Table – Object /ACL/meta/ – Object name => blobUri • BlobInfo Table – BlobUri => FileId – Blob write/read/get • FileInfo Table – FileId => Path, basicInfo • MultiPart Upload Table – uploadId => partInfo

23.对象存储(FDS) BlobStore • BlobManager/FileManager – Blob • magic/data/md5 • offset – • MaxOpenNum/MaxFileSize • Idle Files <=> Busy Files • BlobClient –Blob:'>Blob: