Kubernetes1.5源码分析apiServer之go-restful的使用
唐修能
源码版本
go-restful
简介
go-restful是用于构建REST-style web服务的golang包。
它是出现时因为一个javaer在golang中没找到顺手的REST-based服务构建包，所以 就按照他在java里常用的JAX-RS的设计，在golang中造了一个轮子。
关键组件
1.Route:
路由包含两种，一种是标准JSR311接口规范的实现RouterJSR311,一种是快速路由CurlyRouter。
CurlyRouter支持正则表达式和动态参数，相比RouterJSR11更加轻量级，apiserver中使用的就是这种路由。
一种Route的设定包含：请求方法(http Method)，请求路径(URL Path),输入输出类型(JSON/YAML)以及对应的回掉函数restful.RouteFunction，响应内容类型(Accept)等。
2.WebService：
WebService逻辑上是Route的集合，功能上主要是为一组Route统一设置包括root path,请求响应的数据类型等一些通用的属性。
需要注意的是，WebService必须加入到Container中才能生效。
func (g *APIGroupVersion) InstallREST(container *restful.Container) error {
installer := g.newInstaller()
ws := installer.NewWebService() 。。。
container.Add(ws) return utilerrors.NewAggregate(registrationErrors) }
上面是k8s的REST注册接口，也调用了Container.Add(ws)，才能让这个ws生效。
3.Container：
Container逻辑上是WebService的集合，功能上可以实现多终端的效果。
它包括一组restful.WebService和一个http.ServeMux对象，使用RouteSelector进行请求派发。
例如，下面代码中创建了两个Container，分别在不同的port上提供服务。
该代码是go-restful的example：
package main
import ( "github.com/emicklei/go-restful" "io" "log" "net/http" )
func main() {
ws := new(restful.WebService)
ws.Route(ws.GET("/hello").To(hello)) // ws被添加到默认的container restful.DefaultContainer中
restful.Add(ws)
go func() { // restful.DefaultContainer监听在端口8080上
http.ListenAndServe(":8080", nil) }()
container2 := restful.NewContainer()
ws2 := new(restful.WebService)
ws2.Route(ws2.GET("/hello").To(hello2)) // ws2被添加到container2中
container2.Add(ws2) // container2中监听端口8081
server := &http.Server{Addr: ":8081", Handler: container2}
log.Fatal(server.ListenAndServe()) }
func hello(req *restful.Request, resp *restful.Response) {
io.WriteString(resp, "default world") }
func hello2(req *restful.Request, resp *restful.Response) {
io.WriteString(resp, "second world") }
4.Filter:
Filter用于动态的拦截请求和响应，类似于放置在相应组件前的钩子，在相应组件功能运行前捕获请求或者响应，主要用于记录log，验证，重定向等功能。
go-restful中有三种类型的Filter：
– Container Filter:
运行在Container中所有的WebService执行之前。
// install a (global) filter for the default container (processed before any webservice) restful.Filter(globalLogging)
– WebService Filter：
运行在WebService中所有的Route执行之前。
// install a webservice filter (processed before any route) ws.Filter(webserviceLogging).Filter(measureTime)
– Route Filter:
运行在调用Route绑定的方法之前。
// install 2 chained route filters (processed before calling findUser)
ws.Route(ws.GET("/{user-id}").Filter(routeLogging).Filter(NewCountFilter().routeCounter).To(findUser))
示例
拿用官方提供的例子：
package main
import ( "github.com/emicklei/go-restful" "log" "net/http" )
type User struct { Id, Name string }
type UserResource struct { // normally one would use DAO (data access object)
users map[string]User }
func (u UserResource) Register(container *restful.Container) { // 创建新的WebService
ws := new(restful.WebService) // 设定WebService对应的路径("/users")和支持的MIME类型(restful.MIME_XML/ restful.MIME_JSON)
ws. Path("/users"). Consumes(restful.MIME_XML, restful.MIME_JSON). Produces(restful.MIME_JSON, restful.MIME_XML) // you can specify this per route as well // 添加路由： GET /{user-id} --> u.findUser
ws.Route(ws.GET("/{user-id}").To(u.findUser)) // 添加路由： POST / --> u.updateUser
ws.Route(ws.POST("").To(u.updateUser)) // 添加路由： PUT /{user-id} --> u.createUser
ws.Route(ws.PUT("/{user-id}").To(u.createUser)) // 添加路由： DELETE /{user-id} --> u.removeUser
ws.Route(ws.DELETE("/{user-id}").To(u.removeUser)) // 将初始化好的WebService添加到Container中
container.Add(ws) } // GET http://localhost:8080/users/1 //
func (u UserResource) findUser(request *restful.Request, response *restful.Response) {
id := request.PathParameter("user-id")
usr := u.users[id] if len(usr.Id) == 0 {
response.AddHeader("Content-Type", "text/plain")
response.WriteErrorString(http.StatusNotFound, "User could not be found.") } else {
response.WriteEntity(usr) } } // POST http://localhost:8080/users // <User><Id>1</Id><Name>Melissa Raspberry</Name></User> //
func (u *UserResource) updateUser(request *restful.Request, response *restful.Response) {
usr := new(User)
err := request.ReadEntity(&usr) if err == nil {
u.users[usr.Id] = *usr
response.WriteEntity(usr) } else {
response.AddHeader("Content-Type", "text/plain")
response.WriteErrorString(http.StatusInternalServerError, err.Error()) } } // PUT http://localhost:8080/users/1 // <User><Id>1</Id><Name>Melissa</Name></User> //
func (u *UserResource) createUser(request *restful.Request, response *restful.Response) {
usr := User{Id: request.PathParameter("user-id")}
err := request.ReadEntity(&usr) if err == nil {
u.users[usr.Id] = usr
response.WriteHeader(http.StatusCreated)
response.WriteEntity(usr) } else {
response.AddHeader("Content-Type", "text/plain")
response.WriteErrorString(http.StatusInternalServerError, err.Error()) } } // DELETE http://localhost:8080/users/1 //
func (u *UserResource) removeUser(request *restful.Request, response *restful.Response) {
id := request.PathParameter("user-id") delete(u.users, id) }
func main() { // 创建一个空的Container
wsContainer := restful.NewContainer() // 设定路由为CurlyRouter(快速路由)
wsContainer.Router(restful.CurlyRouter{}) // 创建自定义的Resource Handle(此处为UserResource)
u := UserResource{map[string]User{}} // 创建WebService，并将WebService加入到Container中
u.Register(wsContainer)
log.Printf("start listening on localhost:8080")
server := &http.Server{Addr: ":8080", Handler: wsContainer} // 启动服务
log.Fatal(server.ListenAndServe()) }
上面的示例代码构建RESTful服务，分为几个步骤，apiServer也是类似：
1.创建Container
2.配置Container属性：ServeMux/Router type等
3.创建自定义的Resource Handle，实现Resource相关的处理方式。
4.创建对应Resource的WebService，在WebService中添加响应Route，并将WebService加入到Container中。
5.启动监听服务。
apiServer go-restful使用
Container初始化
apiServer的Container相关的结构是APIContainer。
路径：pkg/genericapiserver/mux/container.go
type APIContainer struct { *restful.Container NonSwaggerRoutes PathRecorderMux SecretRoutes Mux }
而该结构是在GenericAPIServer中被使用，分析过apiServer的启动过程的话，应该对该结构比较熟悉。
type GenericAPIServer struct {
discoveryAddresses DiscoveryAddresses LoopbackClientConfig *restclient.Config
minRequestTimeout time.Duration ...
requestContextMapper api.RequestContextMapper // 这里使用到了restful.Container HandlerContainer *genericmux.APIContainer SecureServingInfo *SecureServingInfo InsecureServingInfo *ServingInfo
effectiveSecurePort, effectiveInsecurePort int ExternalAddress string
storage map[string]rest.Storage Serializer runtime.NegotiatedSerializer Handler http.Handler InsecureHandler http.Handler
apiGroupsForDiscoveryLock sync.RWMutex
apiGroupsForDiscovery map[string]unversioned.APIGroup ... }
而该结构的初始化是在master的初始化过程中进行的。
调用过程： main –> App.Run –> master.Complete.New –> c.Config.GenericConfig.SkipComplete().New()
路径： pkg/genericapiserver/config.go
func (c completedConfig) New() (*GenericAPIServer, error) { if c.Serializer == nil { return nil, fmt.Errorf("Genericapiserver.New() called with config.Serializer == nil") }
s := &GenericAPIServer{
discoveryAddresses: c.DiscoveryAddresses, LoopbackClientConfig: c.LoopbackClientConfig,
legacyAPIGroupPrefixes: c.LegacyAPIGroupPrefixes,
admissionControl: c.AdmissionControl,
requestContextMapper: c.RequestContextMapper, Serializer: c.Serializer,
minRequestTimeout: time.Duration(c.MinRequestTimeout) * time.Second,
enableSwaggerSupport: c.EnableSwaggerSupport, SecureServingInfo: c.SecureServingInfo, InsecureServingInfo: c.InsecureServingInfo, ExternalAddress: c.ExternalAddress,
apiGroupsForDiscovery: map[string]unversioned.APIGroup{},
enableOpenAPISupport: c.EnableOpenAPISupport,
openAPIConfig: c.OpenAPIConfig,
postStartHooks: map[string]postStartHookEntry{}, } // 这里进行了Contianer的初始化
s.HandlerContainer = mux.NewAPIContainer(http.NewServeMux(), c.Serializer) // 添加了DynamicApisDiscovery的
s.installAPI(c.Config)
s.Handler, s.InsecureHandler = c.BuildHandlerChainsFunc(s.HandlerContainer.ServeMux, c.Config) return s, nil }
继续调用mux.NewAPIContainer()接口创建，该接口的两个参数：新建了一个http的ServeMux; 另一个是实现了编解码序列化反序列化的对象
func NewAPIContainer(mux *http.ServeMux, s runtime.NegotiatedSerializer) *APIContainer {
c := APIContainer{ // New一个Container Container: restful.NewContainer(), NonSwaggerRoutes: PathRecorderMux{
mux: mux, }, SecretRoutes: mux, } // 配置http.ServeMux
c.Container.ServeMux = mux
// 配置该Container的路由方式：CurlyRouter 即快速路由
c.Container.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*} // 配置panic产生之后的恢复处理函数
apiserver.InstallRecoverHandler(s, c.Container)
apiserver.InstallServiceErrorHandler(s, c.Container) return &c
}
看下apiserver.InstallRecoverHandler()实现：
func InstallRecoverHandler(s runtime.NegotiatedSerializer, container *restful.Container) {
container.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
logStackOnRecover(s, panicReason, httpWriter) }) } // RecoverHandler changes the default function (logStackOnRecover) to be called // when a panic is detected. DoNotRecover must be have its default value (=false).
func (c *Container) RecoverHandler(handler RecoverHandleFunction) {
c.recoverHandleFunc = handler
}
根据英文注释可以看明白，该RecoverHandler就是在产生panic后会调用的恢复处理函数，默认的调用函数是logStackOnRecover,调用Container.RecoverHandler()后会修改该默认函数，并且Container.DoNotRecover的bool值必须是false才能生效。
apiserver.InstallServiceErrorHandler()接口就不看了，其实就是修改Service Error产生后的错误处理函数，默认是调用writeServiceError()。
到这里Container的初始化基本OK了。
添加WebService
Container已创建并且也进行了初始化。该轮到WebService了，这节会介绍k8s的WebService的创建及添加。
接续上文的Container初始化入口，继续往下看s.installAPI(c.Config):
func (s *GenericAPIServer) installAPI(c *Config) { // 这里原本还有很多routes.Install()函数 // 这些install()貌似和mux有关。 // 而mux就是一个http的多分器，用于派发某个Request路径到对应的http.Handler进行处理 。。。 // 往HandlerContainer中的Container里添加WebService // 该WebService的创建在s.DynamicApisDiscovery()中进行 // 实际上创建的WebService是用于list 该group下的所有versions
s.HandlerContainer.Add(s.DynamicApisDiscovery()) }
先看下WebService的创建接口s.DynamicApisDiscovery()：
路径：pkg/genericapiserver/genericapiserver.go
// DynamicApisDiscovery returns a webservice serving api group discovery. // Note: during the server runtime apiGroupsForDiscovery might change.
func (s *GenericAPIServer) DynamicApisDiscovery() *restful.WebService { return apiserver.NewApisWebService(s.Serializer, APIGroupPrefix, func(req *restful.Request) []unversioned.APIGroup { // 需要加锁 // 接口注释也有说明。因为k8s可以动态加载第三方apiGroups
s.apiGroupsForDiscoveryLock.RLock()
defer s.apiGroupsForDiscoveryLock.RUnlock() // 将apiGroupsForDiscovery中所有的APIGroup按照其名字进行升序排序
sortedGroups := []unversioned.APIGroup{}
groupNames := make([]string, 0, len(s.apiGroupsForDiscovery)) for groupName := range s.apiGroupsForDiscovery {
groupNames = append(groupNames, groupName) }
sort.Strings(groupNames) for _, groupName := range groupNames {
sortedGroups = append(sortedGroups, s.apiGroupsForDiscovery[groupName]) } // 创建切片，并填充各个APIGroup的ServerAddressByClientCIDRs信息
clientIP := utilnet.GetClientIP(req.Request)
serverCIDR := s.discoveryAddresses.ServerAddressByClientCIDRs(clientIP)
groups := make([]unversioned.APIGroup, len(sortedGroups)) for i := range sortedGroups {
groups[i] = sortedGroups[i]
groups[i].ServerAddressByClientCIDRs = serverCIDR
} return groups
}) }
继续深入看apiserver.NewApisWebService()，该接口传入了编解码对象，APIGroup的Prefix，还有一个function。
func NewApisWebService(s runtime.NegotiatedSerializer, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) *restful.WebService { // 用于向后兼容v1.1版本，返回一个空的APIGroup
ss := StripVersionNegotiatedSerializer{s} // 获取支持的媒体类型,比如：application/json，application/yaml
mediaTypes, _ := mediaTypesForSerializer(s) // 构建go-restful的Route处理方法
rootAPIHandler := RootAPIHandler(ss, f) // 创建WebService
ws := new(restful.WebService) // 添加Path
ws.Path(apiPrefix) // API 说明
ws.Doc("get available API versions") // 配置GET("/") 转到rootAPIHandler()接口
ws.Route(ws.GET("/").To(rootAPIHandler). Doc("get available API versions"). Operation("getAPIVersions"). Produces(mediaTypes...). Consumes(mediaTypes...). Writes(unversioned.APIGroupList{})) return ws
}
到这里list某个Group下所有的versions的API已经注册完成了。
这些都不是关键的RESTful API的注册，关键的注册都会在pkg/apiserver/apiserver.go中的InstallREST()接口中进行。
琢磨过apiServer启动流程的同学，应该会知道/api和/apis的注册接口最后都会调用到该接口。
/api的注册接口是pkg/genericapiserver/genericapiserver.go中的InstallLegacyAPIGroup()接口
/apis的注册接口是InstallAPIGroup()。
这两个接口都会调用s.installAPIResources()，最后再调用apiGroupVersion.InstallREST()进行API注册。
流程基本就是这样，接着我们直接进入InstallREST()接口看实现：
func (g *APIGroupVersion) InstallREST(container *restful.Container) error { // 拼装path: "Prefix/Group/Version" // 然后填充并返回一个APIInstaller对象
installer := g.newInstaller() // 创建一个WebService
ws := installer.NewWebService() // 这个是关键，会对各种URL进行注册
apiResources, registrationErrors := installer.Install(ws)
lister := g.ResourceLister if lister == nil {
lister = staticLister{apiResources} } // 增加一个list的API AddSupportedResourcesWebService(g.Serializer, ws, g.GroupVersion, lister) // 将该WebService加入到Container
container.Add(ws) return utilerrors.NewAggregate(registrationErrors) }
前两个调用函数比较简单，这里不进行介绍了。直接进入关键函数installer.Install(ws)：
func (a *APIInstaller) Install(ws *restful.WebService) (apiResources []unversioned.APIResource, errors []error) {
errors = make([]error, 0)
proxyHandler := (&ProxyHandler{
prefix: a.prefix + "/proxy/",
storage: a.group.Storage,
serializer: a.group.Serializer,
mapper: a.group.Context, }) // 将所有的path合成一个切片，并按照升序重新排序
paths := make([]string, len(a.group.Storage)) var i int = 0 for path := range a.group.Storage {
paths[i] = path
i++ }
sort.Strings(paths) for _, path := range paths { // 注册各个URL,关键接口 // 传入的参数：path，rest.Storage，WebService，Handler
apiResource, err := a.registerResourceHandlers(path, a.group.Storage[path], ws, proxyHandler) if err != nil {
errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err)) } // 将所有注册成功的Resource合成一个切片 // 将该切片作为返回值，便于之后的接口注册list Resources的API if apiResource != nil {
apiResources = append(apiResources, *apiResource) } } return apiResources, errors
}
该接口先是遍历所有的path，并升序重新排列，然后循环调用接口注册各个URL的API,并将这些注册成功的APIResource加入到同一个切片中。
我们继续看a.registerResourceHandlers()接口：
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService, proxyHandler http.Handler) (*unversioned.APIResource, error) { ... // 构建creater、lister、deleter、updater、watcher等，其实就是storage
creater, isCreater := storage.(rest.Creater)
namedCreater, isNamedCreater := storage.(rest.NamedCreater)
lister, isLister := storage.(rest.Lister)
getter, isGetter := storage.(rest.Getter)
getterWithOptions, isGetterWithOptions := storage.(rest.GetterWithOptions)
deleter, isDeleter := storage.(rest.Deleter)
gracefulDeleter, isGracefulDeleter := storage.(rest.GracefulDeleter)
collectionDeleter, isCollectionDeleter := storage.(rest.CollectionDeleter)
updater, isUpdater := storage.(rest.Updater)
patcher, isPatcher := storage.(rest.Patcher)
watcher, isWatcher := storage.(rest.Watcher)
_, isRedirector := storage.(rest.Redirector)
connecter, isConnecter := storage.(rest.Connecter)
storageMeta, isMetadata := storage.(rest.StorageMetadata) ... var apiResource unversioned.APIResource // k8s资源分为两类：无namespace的RESTScopeNameRoot; 有namespace的RESTScopeNameNamespace // 在对应的path上添加各类actions，并指定对应的handler。 switch scope.Name() { case meta.RESTScopeNameRoot: // Handle non-namespace scoped resources like nodes.
resourcePath := resource
resourceParams := params
itemPath := resourcePath + "/{name}"
nameParams := append(params, nameParam)
proxyParams := append(nameParams, pathParam)
suffix := "" if hasSubresource {
suffix = "/" + subresource
itemPath = itemPath + suffix
resourcePath = itemPath
resourceParams = nameParams
}
apiResource.Name = path
apiResource.Namespaced = false
apiResource.Kind = resourceKind
namer := rootScopeNaming{scope, a.group.Linker, gpath.Join(a.prefix, resourcePath, "/"), suffix} // Handler for standard REST verbs (GET, PUT, POST and DELETE). // Add actions at the resource path: /api/apiVersion/resource
actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
actions = appendIf(actions, action{"DELETECOLLECTION", resourcePath, resourceParams, namer, false}, isCollectionDeleter) // DEPRECATED
actions = appendIf(actions, action{"WATCHLIST", "watch/" + resourcePath, resourceParams, namer, false}, allowWatchList) // Add actions at the item path: /api/apiVersion/resource/{name}
actions = appendIf(actions, action{"GET", itemPath, nameParams, namer, false}, isGetter) if getSubpath {
actions = appendIf(actions, action{"GET", itemPath + "/{path:*}", proxyParams, namer, false}, isGetter) }
actions = appendIf(actions, action{"PUT", itemPath, nameParams, namer, false}, isUpdater)
actions = appendIf(actions, action{"PATCH", itemPath, nameParams, namer, false}, isPatcher)
actions = appendIf(actions, action{"DELETE", itemPath, nameParams, namer, false}, isDeleter)
actions = appendIf(actions, action{"WATCH", "watch/" + itemPath, nameParams, namer, false}, isWatcher) // We add "proxy" subresource to remove the need for the generic top level prefix proxy. // The generic top level prefix proxy is deprecated in v1.2, and will be removed in 1.3, or 1.4 at the latest. // TODO: DEPRECATED in v1.2.
actions = appendIf(actions, action{"PROXY", "proxy/" + itemPath + "/{path:*}", proxyParams, namer, false}, isRedirector) // TODO: DEPRECATED in v1.2.
actions = appendIf(actions, action{"PROXY", "proxy/" + itemPath, nameParams, namer, false}, isRedirector)
actions = appendIf(actions, action{"CONNECT", itemPath, nameParams, namer, false}, isConnecter)
actions = appendIf(actions, action{"CONNECT", itemPath + "/{path:*}", proxyParams, namer, false}, isConnecter && connectSubpath) break case meta.RESTScopeNameNamespace: ... break default: return nil, fmt.Errorf("unsupported restscope: %s", scope.Name()) } ... // 根据之前生成的actions,进行遍历 // 然后在WebService中添加指定的route for _, action := range actions {
versionedObject := storageMeta.ProducesObject(action.Verb) if versionedObject == nil {
versionedObject = defaultVersionedObject
}
reqScope.Namer = action.Namer
namespaced := "" if apiResource.Namespaced {
namespaced = "Namespaced" }
operationSuffix := "" if strings.HasSuffix(action.Path, "/{path:*}") {
operationSuffix = operationSuffix + "WithPath" } if action.AllNamespaces {
operationSuffix = operationSuffix + "ForAllNamespaces"
namespaced = "" } // 判断action的动作类型 // 生成响应的handler，创建route添加到WebService中 switch action.Verb { case "GET": // Get a resource. var handler restful.RouteFunction // 判断是否有参数 if isGetterWithOptions {
handler = GetResourceWithOptions(getterWithOptions, reqScope) } else {
handler = GetResource(getter, exporter, reqScope) } // 生成处理函数
handler = metrics.InstrumentRouteFunc(action.Verb, resource, handler)
doc := "read the specified " + kind
if hasSubresource {
doc = "read " + subresource + " of the specified " + kind
}
route := ws.GET(action.Path).To(handler). Doc(doc). Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")). Operation("read"+namespaced+kind+strings.Title(subresource)+operationSuffix). Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...). Returns(http.StatusOK, "OK", versionedObject). Writes(versionedObject) if isGetterWithOptions { if err := addObjectParams(ws, route, versionedGetOptions); err != nil { return nil, err
} } if isExporter { if err := addObjectParams(ws, route, versionedExportOptions); err != nil { return nil, err
} }
addParams(route, action.Params)
ws.Route(route) case "LIST": // List all resources of a kind. ... case "PUT": // Update a resource. ... case "PATCH": // Partially update a resource ... case "POST": // Create a resource. ... case "DELETE": // Delete a resource. ... case "DELETECOLLECTION": ... case "WATCH": // Watch a resource. ... case "WATCHLIST": // Watch all resources of a kind. ... case "PROXY": // Proxy requests to a resource. ... case "CONNECT": ... } default: return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb) } // Note: update GetAttribs() when adding a custom handler. } return &apiResource, nil }
首先构建creater、lister、getter、deleter、updater、patcher、watcher，其实他们都是storage，只是对应着对etcd的不同操作。
然后针对所有的action，构建响应的handler。创建对应的route,最后把route添加到service里面。这样就完成了api的注册。
关键的REST API注册基本就这样结束了，除此之外还会有很多别的API的注册：
比如APIGroupVersion.InstallREST()接口中的AddSupportedResourcesWebService(g.Serializer, ws, g.GroupVersion, lister)；
GenericAPIServer.InstallLegacyAPIGroup()接口中的apiserver.AddApiWebService()的调用；
等等。。
其实上面也注册了各种REST API，比如像PodList,ServiceList,ReplicationControllerList等。这些就不深入了，都是大同小异。
类似资料：