Karmada Overrider Policy

1. Overrider Policy介绍及使用

1.1 Overrider Policy介绍

OverridePolicy 和ClusterOverridePolicy定义下发到不通成员集群中不通配置,karmada支持的有：

• ImageOverrider 覆盖工负载的镜像
• CommandOverrider 覆盖工作负载的commands
• ArgsOverrider 覆盖工作负载的 args
• PlaintextOverrider 通用工具，覆盖任何种类的资源

具体字段定义如下：

Spec :

字段	类型	必填	描述	示例
ResourceSelectors	[]ResourceSelector	否	限制了此覆盖策略适用的资源类型。 nil 表示匹配所有资源。（字段结构同PropagationPolicy）
OverrideRules	[]RuleWithCluster	否	定义目标集群上的覆盖规则集合。
TargetCluster	*ClusterAffinity	否	定义了对此覆盖策略应用到成员集群的目标选择。 nil 表示匹配所有集群。（字段同PropagationPolicy）
Overriders	Overriders	否	Overriders 表示将应用于资源的覆盖规则，已弃用，请使用OverrideRules

RuleWithCluster：

字段	类型	必填	描述	示例
TargetCluster	*ClusterAffinity	否	定义了对此覆盖策略应用到成员集群的目标选择。 nil 表示匹配所有集群。（字段同PropagationPolicy）
Overriders	Overriders	是	应用于资源的覆盖规则

Overriders：

字段	类型	必填	描述	示例
ImageOverrider	[]ImageOverrider	否	覆盖镜像的规则
Plaintext	[]PlaintextOverrider	否	覆盖规则
CommandOverrider	[]CommandArgsOverrider	否	容器command覆盖规则
ArgsOverrider	[]CommandArgsOverrider	否	容器arg覆盖规则

ImageOverrider :

字段	类型	必填	描述	示例
Predicate	*ImagePredicate	否	默认为nil,如果资源是Pod, ReplicaSet, Deployment, StatefulSet系统自动检测镜像，如果资源对象有多个容器，所有镜像都将被处理。如果不为空，则只处理匹配到的镜像。
Component	ImageComponent	是	假设镜像组成成分：[registry/]repository[:tag]	Registry，Repository，Tag
Operator	OverriderOperator	是	对镜像进行的操作	add，remove，replace
Value	string	否	当Operator为'add'或'replace'时不能为空，默认为空，当operator为remove时忽略。

ImagePredicate ：

字段	类型	必填	描述	示例
Path	string	是	目标字段的路径	/spec/template/spec/containers/0/image

PlaintextOverrider ：

字段	类型	必填	描述	示例
Path	string		目标字段的路径
Operator	OverriderOperator		对目标字段操作类型	add，remove，replace
Value	apiextensionsv1.JSON	否	应用在目标字段的值，当Operator为remove时，此字段必须为空

CommandArgsOverrider：

字段	类型	必填	描述	示例
ContainerName	string	是	容器名
Operator	OverriderOperator	是	应用在commad/args上的操作	add;remove
Value	[]string	否	应用在command/args上的值，当operator为add时该值append到commad/args，当operator为remove时，该值从command/args移除，如果该值为空command/args维持原状。

1.2 Overrider Policy使用

我们以Plaintext功能使用为例：

假设我们已有一个karmada,管理者两个集群member1和member2, 我们创建了一个名为nginx的deployment：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx

 kubectl --kubeconfig=/etc/karmada/karmada-apiserver.config apply -f nginx-deployment.yaml

创建分发策略：

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member1  #分发到成员集群member1和member2
        - member2
    replicaScheduling:
      replicaDivisionPreference: Weighted #划分副本策略
      replicaSchedulingType: Divided  #调度副本策略
      weightPreference:  
        staticWeightList:  #目标集群静态权重
          - targetCluster:
              clusterNames:
                - member1
            weight: 1
          - targetCluster:
              clusterNames:
                - member2
            weight: 1

 kubectl --kubeconfig=/etc/karmada/karmada-apiserver.config apply -f  nginx-propagation.yaml

创建Overrider Policy修改分发到成员集群memer2的镜像为nginx:1.20.2：

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:  
 name: nginx-override
spec:  
 resourceSelectors: 
 - apiVersion: apps/v1     
   kind: Deployment      
   name: nginx  
 targetCluster:    
  clusterNames:    
  - member2  
 overriders:    
  plaintext:    
  - path: "/spec/template/spec/containers/0/image" #要修改目标字段路径 
    operator: replace   #操作动作
    value: "nginx:1.20.2"  #值

 kubectl --kubeconfig=/etc/karmada/karmada-apiserver.config apply -f  nginx-override.yaml

在集群member2查看结果,镜像覆盖会重建pod：

[root@localhost ~]# kubectl get pods nginx-7658f54f57-glw4f -oyaml|grep image
  - image: nginx:1.20.2
    imagePullPolicy: Always
    image: nginx:1.20.2
    imageID: docker-pullable://nginx@sha256:03f3cb0afb7bd5c76e01bfec0ce08803c495348dccce37bcb82c347b4853c00b

2. Overrider Policy工作原理

我看下Overrider Policy实现的源码，分析下其各个功能工作原理.

我们直接从karmada/pkg/util/overridemanager/overridemanager.go文件中的如下方法看起：

// applyPolicyOverriders applies OverridePolicy/ClusterOverridePolicy overriders to target object
// 应该OverridePolicy/ClusterOverridePolicy覆盖目标对象
func applyPolicyOverriders(rawObj *unstructured.Unstructured, overriders policyv1alpha1.Overriders) error {
    // 1. 应用ImageOverrider，修改目标对象镜像地址
    err := applyImageOverriders(rawObj, overriders.ImageOverrider)
    if err != nil {
        return err
    }
    // patch command
    if err := applyCommandOverriders(rawObj, overriders.CommandOverrider); err != nil {
        return err
    }
    // patch args
    if err := applyArgsOverriders(rawObj, overriders.ArgsOverrider); err != nil {
        return err
    }

    return applyJSONPatch(rawObj, parseJSONPatchesByPlaintext(overriders.Plaintext))
}

2.1 ImageOverrider

karmada假定一个镜像可以被分为三个部分组成：[registry/]repository[:tag]

• registry 例如：k8s.gcr.io，fictional.registry.example:10443
• repository 例如： kube-apiserver，fictional/nginx
• tag 例如：latest，v1.19.1，@sha256:dbcc1c35ac38df41fd2f5e4130b32ffdb93ebae8b3dbe638c23575912276fc9c

详细实现看applyImageOverriders方法：

func applyImageOverriders(rawObj *unstructured.Unstructured, imageOverriders []policyv1alpha1.ImageOverrider) error {
    // 1.遍历定义的所有镜像覆盖规则
    for index := range imageOverriders {
        // 2. 构建变动内容
        patches, err := buildPatches(rawObj, &imageOverriders[index])
        if err != nil {
            klog.Errorf("Build patches with imageOverrides err: %v", err)
            return err
        }

        klog.V(4).Infof("Parsed JSON patches by imageOverriders(%+v): %+v", imageOverriders[index], patches)
        // 3. 修改覆盖目标对象的josn字节流内容
        if err = applyJSONPatch(rawObj, patches); err != nil {
            return err
        }
    }

    return nil
}


// buildPatches parse JSON patches from resource object by imageOverriders
func buildPatches(rawObj *unstructured.Unstructured, imageOverrider *policyv1alpha1.ImageOverrider) ([]overrideOption, error) {
    // 1. 如果覆盖策略中没有设定Predicate，会对Pod, ReplicaSet, Deployment, StatefulSet，job资源自动检测其镜像
    if imageOverrider.Predicate == nil {
        return buildPatchesWithEmptyPredicate(rawObj, imageOverrider)
    }
    // 2. 根据设定的Predicate进行处理
    return buildPatchesWithPredicate(rawObj, imageOverrider)
}

在Predicate设定与否两种情况中我们只看设定了Predicate的实现，其核心内容相差不大：

func buildPatchesWithPredicate(rawObj *unstructured.Unstructured, imageOverrider *policyv1alpha1.ImageOverrider) ([]overrideOption, error) {
    patches := make([]overrideOption, 0)
    // 1. 一顿操作解析出当前object中的将被修改的image
    imageValue, err := obtainImageValue(rawObj, imageOverrider.Predicate.Path)
    if err != nil {
        return nil, fmt.Errorf("failed to obtain imageValue with predicate path(%s), error: %v", imageOverrider.Predicate.Path, err)
    }
    // 2. 构建patch
    patch, err := acquireOverrideOption(imageOverrider.Predicate.Path, imageValue, imageOverrider)
    if err != nil {
        return nil, err
    }

    patches = append(patches, patch)
    return patches, nil
}


func acquireOverrideOption(imagePath, curImage string, imageOverrider *policyv1alpha1.ImageOverrider) (overrideOption, error) {
    if !strings.HasPrefix(imagePath, pathSplit) {
        return overrideOption{}, fmt.Errorf("imagePath should be start with / character")
    }
    // 1. 构建新镜像
    newImage, err := overrideImage(curImage, imageOverrider)
    if err != nil {
        return overrideOption{}, err
    }

    return overrideOption{
        Op:    string(policyv1alpha1.OverriderOpReplace),
        Path:  imagePath,
        Value: newImage,
    }, nil
}

func overrideImage(curImage string, imageOverrider *policyv1alpha1.ImageOverrider) (string, error) {
    // 1. 将旧镜像解析为hostname，repository，tag，digest四个部分
    imageComponent, err := imageparser.Parse(curImage)
    if err != nil {
        return "", fmt.Errorf("failed to parse image value(%s), error: %v", curImage, err)
    }
    // 2. 根据设定的Component修改镜像对应部分
    switch imageOverrider.Component {
    case policyv1alpha1.Registry:
        switch imageOverrider.Operator {
        case policyv1alpha1.OverriderOpAdd:
            imageComponent.SetHostname(imageComponent.Hostname() + imageOverrider.Value)
        case policyv1alpha1.OverriderOpReplace:
            imageComponent.SetHostname(imageOverrider.Value)
        case policyv1alpha1.OverriderOpRemove:
            imageComponent.RemoveHostname()
        }
        return imageComponent.String(), nil
    case policyv1alpha1.Repository:
        switch imageOverrider.Operator {
        case policyv1alpha1.OverriderOpAdd:
            imageComponent.SetRepository(imageComponent.Repository() + imageOverrider.Value)
        case policyv1alpha1.OverriderOpReplace:
            imageComponent.SetRepository(imageOverrider.Value)
        case policyv1alpha1.OverriderOpRemove:
            imageComponent.RemoveRepository()
        }
        return imageComponent.String(), nil
    case policyv1alpha1.Tag:
        switch imageOverrider.Operator {
        case policyv1alpha1.OverriderOpAdd:
            imageComponent.SetTagOrDigest(imageComponent.TagOrDigest() + imageOverrider.Value)
        case policyv1alpha1.OverriderOpReplace:
            imageComponent.SetTagOrDigest(imageOverrider.Value)
        case policyv1alpha1.OverriderOpRemove:
            imageComponent.RemoveTagOrDigest()
        }
        return imageComponent.String(), nil
    }

    // should never reach to here
    return "", fmt.Errorf("unsupported image component(%s)", imageOverrider.Component)
}

2.2 CommandOverrider/ArgsOverrider

CommandOverrider和ArgsOverrider的实现相同，调用了相同的方法buildCommandArgsPatches和applyJSONPatch,我们直接看buildCommandArgsPatches的内容：

// buildCommandArgsPatches build JSON patches for the resource object according to override declaration.
func buildCommandArgsPatches(target string, rawObj *unstructured.Unstructured, commandRunOverrider *policyv1alpha1.CommandArgsOverrider) ([]overrideOption, error) {
    switch rawObj.GetKind() {
    case util.PodKind:
        // 1. 如果覆盖目标对象类型是Pod
        return buildCommandArgsPatchesWithPath(target, "spec/containers", rawObj, commandRunOverrider)
    case util.ReplicaSetKind:
        fallthrough
    case util.DeploymentKind:
        fallthrough
    case util.DaemonSetKind:
        fallthrough
    case util.StatefulSetKind:
         // 2. 如果覆盖目标对象类型是ReplicaSet,Deployment,DaemonSet,StatefulSet
        return buildCommandArgsPatchesWithPath(target, "spec/template/spec/containers", rawObj, commandRunOverrider)
    }
    return nil, nil
}

无论修改上面哪种资源的Command 或 Args 最终都调用了buildCommandArgsPatchesWithPath：

func buildCommandArgsPatchesWithPath(target string, specContainersPath string, rawObj *unstructured.Unstructured, commandRunOverrider *policyv1alpha1.CommandArgsOverrider) ([]overrideOption, error) {
    patches := make([]overrideOption, 0)
    // 1. 从覆盖目标对象中解析出所有containers
    containers, ok, err := unstructured.NestedSlice(rawObj.Object, strings.Split(specContainersPath, pathSplit)...)
    if err != nil {
        return nil, fmt.Errorf("failed to retrieves path(%s) from rawObj, error: %v", specContainersPath, err)
    }
    if !ok || len(containers) == 0 {
        return nil, nil
    }
    klog.V(4).Infof("buildCommandArgsPatchesWithPath containers info (%+v)", containers)
    // 2. 遍历containers，构建新的Command 或 Args，组装patches
    for index, container := range containers {
        if container.(map[string]interface{})["name"] == commandRunOverrider.ContainerName {
            commandArgsPath := fmt.Sprintf("/%s/%d/%s", specContainersPath, index, target)
            commandArgsValue := make([]string, 0)
            var patch overrideOption
            // if target is nil, to add new [target]
            if container.(map[string]interface{})[target] == nil {
                patch, _ = acquireAddOverrideOption(commandArgsPath, commandRunOverrider)
            } else {
                for _, val := range container.(map[string]interface{})[target].([]interface{}) {
                    commandArgsValue = append(commandArgsValue, fmt.Sprintf("%s", val))
                }
                patch, _ = acquireReplaceOverrideOption(commandArgsPath, commandArgsValue, commandRunOverrider)
            }

            klog.V(4).Infof("[buildCommandArgsPatchesWithPath] containers patch info (%+v)", patch)
            patches = append(patches, patch)
        }
    }
    // 3. 抛出patches，在外部执行对目标对象json字节流内容的修改
    return patches, nil
}

2.4 PlaintextOverrider

PlaintextOverrider的实现就比较简单了，根据PlaintextOverrider构建RFC 6902 patch，然后应用修改到目标对象完成修改：

func parseJSONPatchesByPlaintext(overriders []policyv1alpha1.PlaintextOverrider) []overrideOption {
    patches := make([]overrideOption, 0, len(overriders))
    // 1. 根据PlaintextOverrider构建RFC 6902 patch
    for i := range overriders {
        patches = append(patches, overrideOption{
            Op:    string(overriders[i].Operator),
            Path:  overriders[i].Path,
            Value: overriders[i].Value,
        })
    }
    return patches
}