投稿
  1. Blog of Howell首页
  2. CI/CD

k8s Devops-02

LJH • • CI/CD, Kubernetes Devops • 阅读 179

课时11：Jenkins使用BlueOcean创建Jenkinsfile

课时12：Jenkins使用Kubernetes Pod执行构建

课时13：Jenkins配置Kubernetes多集群

课时14：KUBECONFIG文件多集群配置

课时15：Jenkins自动化构建Java应用

课时16：Jenkins自动化构建NodeJS应用

课时17：Jenkins自动化构建及镜像制作建议

课时18：Jenkins生产环境及UAT环境流水线设计

课时19：Jenkins基于角色的账号管理

Jenkins使用BlueOcean创建Jenkinsfile

jenkins构建k8s项目流程：

1、构建K8S集群
2、Jenkins调度K8S API
3、动态生成 Jenkins Slave pod
4、Slave pod 拉取 Git 代码／编译／打包镜像
5、推送到镜像仓库 Harbor
6、Slave 工作完成，Pod 自动销毁
7、部署到测试或生产 Kubernetes平台

在此之前我们要先在gitlab的kubernetes-guide组中创建适用于java项目基于BlueOcean的jenkinsfile（spring-cloud-jenkinsfile）的project项目以及代码项目的project（spring-cloud-eureka），将这个目录下的项目上传到我们的spring-cloud-eureka项目中

k8s Devops-02
k8s Devops-02
k8s Devops-02

在github项目中进入到spring-cloud-eureka目录将eureka项目推送上去

k8s Devops-02

添加一个Dockerfile

# 获取github项目
git clone https://github.com/gongchangwangpi/spring-cloud-demo2.git
cd spring-cloud-demo2/spring-cloud-eureka
# 复制到本地gitlab
git clone git@gitlab.test.com:kubernetes-guide/spring-cloud-eureka.git
cd spring-cloud-eureka
cp -r src/ pom.xml spring-cloud-eureka/
cd spring-cloud-eureka/
# 添加Dockerfile
cat < EOF >>  Dockerfile
# openJDK
FROM maven:3.5.3
COPY target/*.jar /opt
EXPOSE 8080
EOF
推送到本地gitlab
git add .
git commit -m "push project"
git push -u origin master

查看推送结果

k8s Devops-02

创建一个spring-cloud-jenkinsfile的BlueOcean

注意：使用BlueOcean创建Jenkinsfile时，任何步骤不能写中文

k8s Devops-02
k8s Devops-02

将gitlab地址复制到连接git仓库的URL中，同时将密钥添加到github中的SSH keys中，这样可以在gitlab私有仓库中拉取代码

k8s Devops-02

使用BlueOcean创建Jenkinsfile，代理这里不支持填写kubernetes，可以修改成any，稍后在jenkinsfile中修改成kubernetes

k8s Devops-02

因为BlueOcean会报错，所以修改成any

k8s Devops-02

添加pulling code / Git步骤

k8s Devops-02

pulling code by trigger / Git 如果是自动触发代码，这里有一个手动选择的变量Branch是获取不到的，这样使用gitlab自动构建产生的分支变量pulling code by trigger，Branch中写env.gitlabBranch，一会儿添加一个when条件，使流水线分别支持手动选择分支和自动（gitlab所有分支）获取代码

k8s Devops-02

继续添加步骤pulling code by trigger / Run arbitrary Pipeline script如果此步骤BRANCH为空，则手动赋值BRANCH = env.gitlabBranch

k8s Devops-02

initConfiguration / Run arbitrary Pipeline script 步骤可以根据分支生成一些提交信息，生成全局变量信息

k8s Devops-02
CommitID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
CommitMessage = sh(returnStdout: true, script: "git log -1 --pretty=format:'%h : %an  %s'").trim()
def curDate = sh(script: "date '+%Y%m%d-%H%M%S'", returnStdout: true).trim()
TAG = curDate[0..14] + "-" + CommitID + "-" + BRANCH

将CommitID变量作为之后镜像推送tag赋值变量

k8s Devops-02

TAG = curDate[0..14] + "-" + CommitID + "-" + BRANCH代表由时间前14位+CommitID+分支变量作为镜像ID

添加到kubernetes变量中

k8s Devops-02

因为实在容器中进行构建，添加Build and test / Run build steps in a container步骤，容器名称为build即可，子步骤中使用Shell Script，因为不同java项目构建命令可能不同，所以将构建命令转到项目变量${BUILD_COMMAND}中

k8s Devops-02
sh """
  echo 'building...'
  ${BUILD_COMMAND}
"""

Build and test同步阶段进行代码扫描操作Scan Code / Shell Script，因为没有安装代码扫描软件，所以这里简单执行一个ls

k8s Devops-02
sh """
  echo "scanning code"
  ls
"""

添加Build docker image / Run build steps in a container，Name叫做docker即可，并添加Shell Script子步骤

k8s Devops-02

Shell Script子步骤中：

k8s Devops-02
sh """
  docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
  docker login -u ${Username} -p ${Password} ${HARBOR_ADDRESS}
  docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""

设置了诸多后续设置的变量，因为docker镜像仓库是根据阿里云容器镜像服务推出，这里的变量对应需要解释一下

下面的howell-kubernetes在阿里云叫做的命名空间，在dockerhub中可以叫镜像仓库路径，spring-cloud-demo-eureka在阿里云中叫做镜像仓库，在dockerhub中可以叫做镜像名称（名字不一样，但本质一样），模拟阿里云推送过程，设置变量

k8s Devops-02
k8s Devops-02

添加Deploy / Run build steps in a container步骤名称为kubectl，子步骤Shell Script为：

k8s Devops-02
sh """
  cat ${KUBECONFIG_PATH} > /tmp/1.yaml
  /usr/local/bin/kubectl config use-context ${CLUSTER} --kubeconfig=/tmp/1.yaml
  export KUBECONFIG=/tmp/1.yaml
  /usr/local/bin/kubectl set image ${DEPLOY_TYPE} -l ${DEPLOY_LABEL} ${CONTAINER_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n ${NAMESPACE}
"""

上述命令步骤有切换到此集群环境，然后设置部署控制器类型下的指定容器名称的image进行更新，因为一个镜像可能被多个部署文件使用，所以使用-l添加部署范围

最后保存，添加一个java-pipline的分支，用来部署java项目

k8s Devops-02

查看并修改完善自动生成的jenkinsfile

k8s Devops-02

Jenkins使用Kubernetes Pod执行构建

这里添加了kubernetes相关配置

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes-default'
      slaveConnectTimeout 1200
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
    - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
      # 动态生成 Jenkins Slave pod，jnlp镜像为master创建的slave，负责临时产生相关任务命令，JNLP方式连接salve，不需要master必须能够ssh连接到slave，只需要两者能够ping通即可。这种连接方式的slave还可以作为服务运行在slave的机器上，开始pod会被创建，最后pod会被消除
      image: 'jenkins/jnlp-slave:4.13.3-1-jdk11'
      name: jnlp
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false        
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      # build镜像承担项目构建任务
      image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
      imagePullPolicy: "IfNotPresent"
      name: "build"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/root/.m2/"
          name: "volume-maven-repo"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      # kubectl镜像承担更新镜像任务，因为要在docker中创建docker，所以将宿主机的docker.sock通过volume-docker方式挂载到容器中，并且将kubeconfig配置到容器中
      image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
      imagePullPolicy: "IfNotPresent"
      name: "kubectl"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-docker"
          readOnly: false
        - mountPath: "/mnt/.kube/"
          name: "volume-kubeconfig"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      # docker容器承担推送镜像任务，需要使用到docker命令，所以需要挂载docker.sock
      image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
      imagePullPolicy: "IfNotPresent"
      name: "docker"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-docker"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
  restartPolicy: "Never"
  nodeSelector:
    build: "true"
  securityContext: {}
  volumes:
    - hostPath:
        path: "/var/run/docker.sock"
      name: "volume-docker"
    - hostPath:
        path: "/usr/share/zoneinfo/Asia/Shanghai"
      name: "volume-2"
    - hostPath:
        path: "/etc/hosts"
      name: "volume-hosts"
    - name: "volume-maven-repo"
      hostPath:
        path: "/opt/m2"
    - name: "volume-kubeconfig"
      secret:
        secretName: "multi-kube-config"
'''	
    }
  }

  stages {
    stage('pulling code') {
      parallel {
        stage('pulling code') {
          when {
            expression {
              env.gitlabBranch == null
            }
          }
          steps {
            git(url: "${REPO_URL}", branch: "${BRANCH}", changelog: true, credentialsId: 'de65d2d6-3a5a-49b3-ab3f-bac850d23f21')
          }
        }

        stage('pulling code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
            }
          }
          steps {
            git(url: "${REPO_URL}", branch: env.gitlabBranch, changelog: true, credentialsId: 'de65d2d6-3a5a-49b3-ab3f-bac850d23f21')
            script {
              BRANCH = env.gitlabBranch
            }

          }
        }

      }
    }

    stage('initConfiguration') {
      steps {
        script {
          CommitID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
          CommitMessage = sh(returnStdout: true, script: "git log -1 --pretty=format:'%h : %an  %s'").trim()
          def curDate = sh(script: "date '+%Y%m%d-%H%M%S'", returnStdout: true).trim()
          TAG = curDate[0..14] + "-" + CommitID + "-" + BRANCH
        }

      }
    }

    stage('Build and test') {
      parallel {
        stage('Build and test') {
          steps {
            container(name: 'build') {
              sh """
              echo "building..."
              ${BUILD_COMMAND}
              """
            }

          }
        }

        stage('Scan Code') {
          steps {
            sh """
            echo "scanning code"
            ls
            """
          }
        }

      }
    }

    stage('Build docker image') {
      steps {
        withCredentials([usernamePassword(credentialsId: 'REGISTRY_USER', passwordVariable: 'Password', usernameVariable: 'Username')]) {
          container(name: 'docker') {
            sh """
            docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
            docker login -u ${Username} -p ${Password} ${HARBOR_ADDRESS}
            docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
            """
          }
        }

      }
    }

    stage('Deploy') {
      steps {
        container(name: 'kubectl') {
          sh """
          cat ${KUBECONFIG_PATH} > /tmp/1.yaml
          /usr/local/bin/kubectl config use-context ${CLUSTER} --kubeconfig=/tmp/1.yaml
          export KUBECONFIG=/tmp/1.yaml
          /usr/local/bin/kubectl set image ${DEPLOY_TYPE} -l ${DEPLOY_LABEL} ${CONTAINER_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n ${NAMESPACE}
          """
        }

      }
    }

  }
  environment {
    CommitMessage = ''
    CommitID = ''
    TAG = ''
  }
}

需要注意一下等会在变量参数中需要添加的变量有：

# =============需要在项目中定义的参数变量===============
# 项目获取相关
REPO_URL（Hidden Parameter）= git@gitlab.test.com:kubernetes-guide/spring-cloud-eureka.git
BRANCH（List Git branches）：
Repository URL=git@gitlab.test.com:kubernetes-guide/spring-cloud-eureka.git
Credentials=gitlab ssh key
Parameter Type=Branch

# 项目构建相关
BUILD_COMMAND（String Parameter）= mvn clean package -DiskipTests

# 容器镜像相关
HARBOR_ADDRESS（Hidden Parameter）=registry.cn-beijing.aliyuncs.com
REGISTRY_DIR（Hidden Parameter）=howell-kubernetes
IMAGE_NAME（Hidden Parameter）=spring-cloud-demo-eureka

# k8s部署更新相关
KUBECONFIG_PATH（Hidden Parameter）=/mnt/.kube/multi-cluster.yaml
CLUSTER（Hidden Parameter）=test
DEPLOY_TYPE（Hidden Parameter）=deployment
DEPLOY_LABEL（Hidden Parameter）= app=spring-cloud-demo-eureka
CONTAINER_NAME（Hidden Parameter）= spring-cloud-demo-eureka
NAMESPACE（Hidden Parameter）=java-test

# 添加一个选择部署项，决定是否直接部署
DEPLOY（Choice Parameter）=
true
false


# =============其他变量=============
# 部分变量已经在Jenkins中内置：$(JENKINS_SECRET) $(JENKINS_NAME) ${Username} ${Password} kubernetes-default
# 脚本中已经定义的全局变量无需再定义：CommitID CommitMessage TAG

创建一个新的视图

k8s Devops-02
k8s Devops-02
k8s Devops-02

新建一个item，添加到当前视图

k8s Devops-02

配置流水线选择Pipeline script from SCM，选择spring-cloud-jenkinsfile项目下对应分支java-pipeline下的jenkinsfile文件

k8s Devops-02

将之前的变量全部添加进去

使用List Git branches添加BRANCH变量

k8s Devops-02
k8s Devops-02

使用Hidden Parameter添加REPO_URL变量

k8s Devops-02

使用String Parameter添加BUILD_COMMAND变量

k8s Devops-02

使用Hidden Parameter添加HARBOR_ADDRESS变量

k8s Devops-02

使用Hidden Parameter添加REGISTRY_DIR变量

k8s Devops-02

使用Hidden Parameter添加IMAGE_NAME变量

k8s Devops-02

使用Hidden Parameter添加CLUSTER变量

k8s Devops-02

使用Hidden Parameter添加KUBECONFIG_PATH变量

k8s Devops-02

使用Hidden Parameter添加DEPLOY_TYPE变量

k8s Devops-02

使用Hidden Parameter添加DEPLOY_LABEL变量

k8s Devops-02

使用Hidden Parameter添加CONTAINER_NAME变量

k8s Devops-02

使用Hidden Parameter添加NAMESPACE变量

k8s Devops-02

使用Choice Parameter添加DEPLOY变量

k8s Devops-02

Jenkins配置Kubernetes多集群

jenkins链接k8s集群

k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02

创建kubernetes凭据

k8s Devops-02
k8s Devops-02

在全局system中配置一个credentials：
新版Jenkins：Manage Jenkins -> Credentials -> System -> Global credentials -> Add Credentials

k8s Devops-02

在k8s-master01中使用openssl生成我们的pkcs文件

cd /etc/kubernetes/pki
openssl pkcs12 -export -out /tmp/default.pfx -inkey admin-key.pem -in admin.pem -certfile ca.pem

将default.pfx导入到我们的Credentials中

k8s Devops-02

开启jenkins jnlp slave和master端口，一般都设置成50000

k8s Devops-02
k8s Devops-02

如果要配置多个集群，按照此方法配置多个名称的jenkins即可

KUBECONFIG文件多集群配置

[root@k8s-master01 ~]# cd /etc/kubernetes/pki
[root@k8s-master01 pki]# cp ~/.kube/config ./multi-cluster.yaml

# 添加test集群
[root@k8s-master01 pki]# kubectl config set-cluster test --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:8443 --kubeconfig=multi-cluster.yaml 
Cluster "test" set.

[root@k8s-master01 pki]# kubectl config set-credentials test-admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=multi-cluster.yaml 
User "test-admin" set.

[root@k8s-master01 pki]# kubectl config set-context test --cluster=test --user=test-admin --kubeconfig=multi-cluster.yaml 
Context "test" created.

# 添加uat集群
kubectl config set-cluster uat --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.100:8443 --kubeconfig=multi-cluster.yaml

kubectl config set-credentials uat-admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=multi-cluster.yaml

kubectl config set-context uat --cluster=uat --user=uat-admin --kubeconfig=multi-cluster.yaml

# 切换测试：
kubectl config use-context test --kubeconfig=/etc/kubernetes/pki/multi-cluster.yaml
kubectl config use-context uat --kubeconfig=/etc/kubernetes/pki/multi-cluster.yaml

# 生成secret，之后会挂载到k8s创建的容器中：
kubectl create secret generic multi-kube-config --from-file=/etc/kubernetes/pki/multi-cluster.yaml

Jenkins自动化构建Java应用

想要拉取代码，我们还需要docker仓库的密码认证，在所在的名称空间创建一个dockerconfigjson

kubectl create ns java-test

kubectl create secret docker-registry myregistrykey --docker-server=registry.cn-beijing.aliyuncs.com --docker-username=XXXXX --docker-password=XXXXXX --docker-email=XXXXX@163.com -n java-test

部署我们的eureka服务

vim deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: spring-cloud-demo-eureka
  name: spring-cloud-demo-eureka
  namespace: java-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-cloud-demo-eureka
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: spring-cloud-demo-eureka
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - java -jar /opt/*.jar
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: registry.cn-beijing.aliyuncs.com/howell-kubernetes/spring-cloud-demo-eureka:20230928-210151-41fc05a-master
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - pgrep java
          failureThreshold: 2
          initialDelaySeconds: 80
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: spring-cloud-demo-eureka
        ports:
        - containerPort: 8761
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 80
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8761
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1000m
            memory: 1000Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
      imagePullSecrets:
      - name: myregistrykey
      nodeSelector:
        build: "true"
      restartPolicy: Always
      serviceAccount: java-cluster
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: spring-cloud-demo-eureka-svc
  name: spring-cloud-demo-eureka-svc
  namespace: java-test
spec:
  ports:
  - port: 8761
    protocol: TCP
    targetPort: 8761
  selector:
    app: spring-cloud-demo-eureka
  type: ClusterIP

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spring-cloud-demo-eureka
  namespace: java-test
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: eureka.test.com
    http:
      paths:
      - backend:
          service:
            name: spring-cloud-demo-eureka-svc
            port:
              number: 8761
        path: /
        pathType: Prefix
k8s Devops-02

开始构建：

k8s Devops-02
k8s Devops-02

新pod自动创建，旧pod被删除

k8s Devops-02

kubectl -n java-test get deployments.apps spring-cloud-demo-eureka -o yaml | grep image

对比推送的和目前部署的镜像一致即成功更新容器景象版本

k8s Devops-02
k8s Devops-02

Jenkins自动化构建NodeJS应用

在原有的gitlab项目中新建一个分支，从java-pipeline项目中复制

k8s Devops-02
k8s Devops-02

修改一下编译镜像，网速乐观情况下也可以使用docker官网的node:4.2.3

k8s Devops-02

自动构建NodeJS应用：https://github.com/selaworkshops/npm-demo-app

将镜像下载下来上传到本地的gitlab，按照项目所要求的创建Dockerfile

cd npm-demo-app

vim Dockerfile

FROM node:4.2.3

WORKDIR /app

COPY package.json /app/
# RUN npm install 编译操作独立出去
COPY . /app

EXPOSE 3000

ENTRYPOINT ["npm", "start"]
k8s Devops-02
k8s Devops-02

创建新的项目，将代码上传上去

cd ..
git clone git@gitlab.test.com:kubernetes-guide/nodejs-demo.git
cp -rp npm-demo-app/* nodejs-demo/
cd nodejs-demo
git commit -am "first commit nodejs"
git push -u origin master
k8s Devops-02

在jenkins界面创建一个nodejs-demo，基于spring-cloud-eureka复制即可

k8s Devops-02

对相关参数做出相应修改

k8s Devops-02
k8s Devops-02
k8s Devops-02

创建一个名称为nodejs-demo镜像仓库，然后修改参数

k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02

执行build，不进行deploy，只进行镜像推送

k8s Devops-02
k8s Devops-02
k8s Devops-02

测试镜像：

docker run --name=nodejs-test-docker -it -p 3000:3000 --rm registry.cn-beijing.aliyuncs.com/howell-kubernetes/nodejs-demo:20231004-173620-2f05435-master
k8s Devops-02

测试本地访问

k8s Devops-02

使用首次推送的镜像进行部署：

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: nodejs-test
spec: {}
status: {}

---
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: myregistrykey
  namespace: nodejs-test
stringData:
  .dockerconfigjson: '{"auths":{"registry.cn-beijing.aliyuncs.com":{"username":"燎原xxxxx","password":"xxxxxxxx","email":"ljhxxxxxxx@163.com","auth":"54eO5Y6f5LmL54GrbGpoOkxqaDcyMTEyMg=="}}}'
type: kubernetes.io/dockerconfigjson

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations: {}
  labels:
    app: nodejs-demo
  name: nodejs-demo
  namespace: nodejs-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nodejs-demo
  template:
    metadata:
      labels:
        app: nodejs-demo
    spec:
      affinity: {}
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: registry.cn-beijing.aliyuncs.com/howell-kubernetes/nodejs-demo:20231004-173620-2f05435-master
        imagePullPolicy: IfNotPresent
        lifecycle: {}
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 3000
          timeoutSeconds: 2
        name: nodejs-demo
        ports:
        - containerPort: 3000
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 3000
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1001m
            memory: 1073Mi
          requests:
            cpu: 10m
            memory: 10Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
      imagePullSecrets:
      - name: myregistrykey
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nodejs-demo
  name: nodejs-demo
  namespace: nodejs-test
spec:
  ports:
  - name: port3000
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: nodejs-demo
  type: ClusterIP

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nodejs-demo
  namespace: nodejs-test
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: nodejs.test.com
    http:
      paths:
      - backend:
          serviceName: nodejs-demo
          servicePort: 3000
        path: /

修改本地hosts文件，添加域名nodejs.test.com进行访问

k8s Devops-02

测试发版：开发修改代码重新提交版本

修改一下title进行发版

k8s Devops-02
k8s Devops-02

发版成功：

k8s Devops-02

Jenkins自动化构建及镜像制作建议

在生产环境中，我们可以将构建镜像和缓存目录进一步做成变量独立出去，建议参数化统一jenkinsfile

建议源码和依赖分开为两层容器层先铺一层依赖，然后再拷贝源码，在之后重新获取镜像时，可以少拉取依赖层的步骤，加快部署效率

Jenkins生产环境及UAT环境流水线设计

生产环境的发布流程往往是选择我们测试环境的镜像发版到对应生产环境中实现发版

测试版本->容器镜像发版->测试环境版本更新部署->生产环境记录容器镜像tag->正式环境选择镜像部署

实验中我们使用一个新的集群UAT创建一个新的名称空间uat来对nodejs项目进行线上部署和版本更新

在新的k8s集群创建一个nodejs-uat的名称空间，将Deployment、Service、Secret，ingress（新添加域名uat-nodejs.test.com）复制到此名称空间，名字相同即可，因为实验环境生产环境一般都会保持一致

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: nodejs-demo
  name: nodejs-demo
  namespace: nodejs-uat
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nodejs-demo
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodejs-demo
    spec:
      affinity: {}
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: registry.cn-beijing.aliyuncs.com/howell-kubernetes/nodejs-demo:20231004-183115-5c08c89-master
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 3000
          timeoutSeconds: 2
        name: nodejs-demo
        ports:
        - containerPort: 3000
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 3000
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1001m
            memory: 1073Mi
          requests:
            cpu: 10m
            memory: 10Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: myregistrykey
      restartPolicy: Always
      schedulerName: default-scheduler
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-10-05T11:02:09Z"
  labels:
    app: nodejs-demo
  name: nodejs-demo
  namespace: nodejs-uat
spec:
  ports:
  - name: port3000
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: nodejs-demo
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: nodejs-demo
  namespace: nodejs-uat
spec:
  rules:
  - host: nodejs-uat.test.com
    http:
      paths:
      - backend:
          service:
            name: nodejs-demo
            port:
              number: 3000
        path: /
        pathType: ImplementationSpecific

访问：

k8s Devops-02

jenkins创建新的视图和job，复制原先的nodejs项目，然后修改相关参数

k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02
k8s Devops-02

添加一个镜像tag参数，Fallback设置为return 'error'，参考变量为REGISTRY_DIR,IMAGE_NAME

def get_tags = [ "bash", "-c", "aliyun cr  GetRepoTags --RepoNamespace=${REGISTRY_DIR} --RepoName=${IMAGE_NAME} | jq '.data.tags[].tag' -r" ]

return get_tags.execute().text.tokenize('\n')
k8s Devops-02
k8s Devops-02
k8s Devops-02

uat流水线脚本内容

k8s Devops-02
pipeline {
   agent any

   stages {
      stage('Hello') {
         steps {
            sh """
               echo ${IMAGE_TAG}
               kubectl config use-context --kubeconfig=${KUBECONFIG_PATH} ${CLUSTER}
               kubectl --kubeconfig=${KUBECONFIG_PATH} set image ${DEPLOY_TYPE} -l ${DEPLOY_LABEL} ${CONTAINER_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${IMAGE_TAG} -n ${NAMESPACE}
               kubectl --kubeconfig=${KUBECONFIG_PATH} get po  -l ${DEPLOY_LABEL} -n ${NAMESPACE} -w
            """
         }
      }
   }
}

成功获取到镜像

k8s Devops-02

将kubectl命令和multi-cluster.yaml配置文件传送到jenkins所在服务器中

配置文件无需挂载到镜像中，所以直接根据参数传输到/mnt/.kube/multi-cluster.yaml即可

k8s Devops-02

scp /etc/kubernetes/pki/multi-cluster.yaml root@k8s-master02:/mnt/.kube/multi-cluster.yaml

kubectl get po -n nodejs-uat --kubeconfig=/etc/kubernetes/pki/multi-cluster.yaml -o yaml | grep image:

k8s Devops-02
k8s Devops-02

发版成功：

k8s Devops-02

Jenkins基于角色的账号管理

需要使用一个插件：Role-based Authorization Strategy 如果没有自行安装一下

k8s Devops-02

装好之后在manager jenkins中找到security，将Authentication手去按策略修改为role-based Strategy模式

k8s Devops-02
k8s Devops-02

manage Jenkins会出现一个新的选项manage and assign roles

k8s Devops-02

添加一个test视图，将test后缀的项目添加到视图中此时可以发现，不通项目在不同视图中

k8s Devops-02

我们添加两个用户，一个叫test，一个叫uat，test用户可以访问”-test“结尾的项目，uat同理，进入manage and assign roles

k8s Devops-02

添加一个全局角色anon，配置Overall/Read permission，用于全局查看

k8s Devops-02

添加 Overall（Read）权限，保存

k8s Devops-02

在Assign Roles配置中Anonymous添加anon权限

k8s Devops-02

添加一个项目角色test-user，正则匹配.*-test

k8s Devops-02

作为一个项目构建用户，一般只拥有build构建操作以及看日志的操作，提供其Credentials（View）Job（Discover、Read、Workspace）权限

k8s Devops-02

在manage Jenkins中创建一个用户

k8s Devops-02
k8s Devops-02

回到manage and assign roles，选择Assign Roles配置，添加test到item角色，保存

k8s Devops-02
k8s Devops-02

登陆test用户

k8s Devops-02

获取到-test项目，其他项目无法获取到，尝试build操作，可以进行build和查看日志，但是没有config编辑操作

k8s Devops-02
k8s Devops-02
k8s Devops-02

配置一个test项目的管理员用户，提供Credentials（Create、Update、View）、Job（Configure、Create、Delete、Discover、Read、Workspace）权限，保存

k8s Devops-02

创建一个test-manager用户

k8s Devops-02

在Assign Role中绑定权限，save保存

k8s Devops-02

登陆test-manager用户

k8s Devops-02

在项目中我们可以看到只有对-test项目只有配置和删除权限

k8s Devops-02
k8s Devops-02

发布者：LJH，转发请注明出处：https://www.ljh.cool/38348.html

赞 (0)
LJHLJH
0 0
上一篇 2023年9月18日 上午11:30
下一篇 2023年10月8日 下午9:46

相关推荐

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注