
一般构建的流程如下:
1) 在 GitLab 中创建对应的项目;
2) 配置 Jenkins 集成 Kubernetes 集群,后期 Jenkins 的 Slave 将为在 Kubernetes 中动态创建的 Slave;
3) Jenkins 创建对应的任务(Job),集成该项目的 Git 地址和 Kubernetes 集群;
4) 开发者将代码提交到 GitLab;
5) 如有配置钩子,推送(Push)代码会自动触发 Jenkins 构建,如没有配置钩子,需要手动构建;
6) Jenkins 控制 Kubernetes(使用的是 Kubernetes 插件)创建 Jenkins Slave(Pod 形式);
7) Jenkins Slave 根据流水线(Pipeline)定义的步骤执行构建;
8) 通过 Dockerfile 生成镜像;
9) 将镜像提送(Push)到私有 Harbor(或者其它的镜像仓库);
10) Jenkins 再次控制 Kubernetes 进行最新的镜像部署;
11) 流水线结束删除 Jenkins Slave。
一.创建Jenkinsfile流水线(java)
1.通过blue ocean设计框架
我们可以通过blue ocean设计框架,通过导出的jenkinsfile来添加需要的内容,以节省时间:

2.定义jenkinsfile
在java项目代码仓库添加一个jenkinsfile文件:

注意文件名称为:Jenkinsfile(J是大写)
复制上步骤导出的框架和内容:
pipeline {
agent {
kubernetes {
cloud 'study-kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
imagePullPolicy: "IfNotPresent"
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/root/.m2/"
name: "cachedir"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: "IfNotPresent"
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: "IfNotPresent"
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
}
stages {
stage('Pulling Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git(changelog: true, poll: true, url: 'git@192.168.0.111:root/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key')
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git(url: 'git@192.168.0.111:root/spring-boot-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key')
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
}
}
stage('Building') {
steps {
container(name: 'build') {
sh """
mvn clean install -DskipTests
ls target/*
"""
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('HARBOR_ACCOUNT')
}
steps {
container(name: 'docker') {
sh """
echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8s') {
environment {
MY_KUBECONFIG = credentials('study-kubernetes')
}
steps {
container(name: 'kubectl'){
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.0.110"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "spring-boot-project"
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH')
}
}
3.jenkinsfile详解
(1).顶层agent
首先是顶层的Agent,定义的是Kubernetes 的Pod 作为Jenkins 的Slave:
agent {
# 定义使用Kubernetes 作为agent
kubernetes {
# 选择的云为之前配置的名字
cloud 'kubernetes-study'
slaveConnectTimeout 1200
# 将workspace 改成hostPath,因为该Slave 会固定节点创建,如果有存储可用,可以
改成PVC 的模式
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace",
readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
# jnlp 容器,和Jenkins 主节点通信
- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# build 容器,包含执行构建的命令,比如Java 的需要mvn 构建,就可以用一个maven 的
镜像
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
# 使用Maven 镜像,包含mvn 工具。NodeJS 可以用node 的镜像
image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
imagePullPolicy: "IfNotPresent"
# 容器的名字,流水线的stage 可以直接使用该名字
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
# Pod 单独创建了一个缓存的volume,将其挂载到了maven 插件的缓存目录,默认是
/root/.m2
- mountPath: "/root/.m2/"
name: "cachedir"
readOnly: false
# 发版容器,因为最终是发版至Kubernetes 的,所以需要有一个kubectl 命令
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
# 镜像的版本可以替换为其它的版本,也可以不进行替换,因为只执行set 命令,所以版本
是兼容的
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: "IfNotPresent"
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# 用于生成镜像的容器,需要包含docker 命令
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-
git"
imagePullPolicy: "IfNotPresent"
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# 由于容器没有启动docker 服务,所以将宿主机的docker 经常挂载至容器即可
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
# 固定节点部署
nodeSelector:
build: "true"
securityContext: {}
volumes:
# Docker socket volume
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
# 缓存目录
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
之后看一下Jenkinsfile 最后的环境变量和parameters 的配置:
# 定义一些全局的环境变量
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "CHANGE_HERE_FOR_YOUR_HARBOR_URL" # Harbor 地址
REGISTRY_DIR = "kubernetes" # Harbor 的项目目录
IMAGE_NAME = "spring-boot-project" # 镜像的名称
NAMESPACE = "kubernetes" # 该应用在Kubernetes 中的命名空间
TAG = "" # 镜像的Tag,在此用BUILD_TAG+COMMIT_ID 组成
}
parameters {
# 之前讲过一些choice、input 类型的参数,本次使用的是GitParameter 插件
# 该字段会在Jenkins 页面生成一个选择分支的选项
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue:
'', description: 'Branch for build and deploy', name: 'BRANCH',
quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE',
tagFilter: '*', type: 'PT_BRANCH')
}
(2).拉取代码判断
接下来是拉代码的 stage,这个 stage 是一个并行的 stage,因为考虑了该流水线是手动触
发还是触发:
stage('Pulling Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
# 假如env.gitlabBranch 为空,则该流水线为手动触发,那么就会执行该
stage,如果不为空则会执行同级的另外一个stage
env.gitlabBranch == null
}
}
steps {
# 这里使用的是git 插件拉取代码,BRANCH 变量取自于前面介绍的parameters
配置
# git@xxxxxx:root/spring-boot-project.git 代码地址
# credentialsId: 'gitlab-key',之前创建的拉取代码的key
git(changelog: true, poll: true, url: 'git@xxxxxx:root/spring-
boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key')
script {
# 定义一些变量用于生成镜像的Tag
# 获取最近一次提交的Commit ID
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --
pretty=format:'%h'").trim()
# 赋值给TAG 变量,后面的docker build 可以取到该TAG 的值
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${BRANCH}, Commit ID is
${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
# 如果env.gitlabBranch 不为空,说明该流水线是通过webhook 触发,则此
时执行该stage,上述的stage 不再执行。此时BRANCH 变量为空
env.gitlabBranch != null
}
}
steps {
# 以下配置和上述一致,只是此时branch: env.gitlabBranch 取的值为
env.gitlabBranch
git(url: 'git@xxxxxxxxxxx:root/spring-boot-project.git',
branch: env.gitlabBranch, changelog: true, poll: true, credentialsId:
'gitlab-key')
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --
pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${env.gitlabBranch}, Commit ID is
${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
}
}
(3).编译代码
代码拉下来后,就可以执行构建命令,由于本次实验是Java 示例,所以需要使用mvn 命令进行构建:
stage('Building') {
steps {
# 使用Pod 模板里面的build 容器进行构建
container(name: 'build') {
sh """
# 编译命令, 需要根据自己项目的实际情况进行修改,可能会不一致
mvn clean install -DskipTests
# 构建完成后,一般情况下会在target 目录下生成jar 包
ls target/*
"""
}
}
}
(4).生成镜像并推送到镜像仓库
生成编译产物后,需要根据该产物生成对应的镜像,此时可以使用Pod 模板的docker 容器:
stage('Docker build for creating image') {
# 首先取出来HARBOR 的账号密码
environment {
HARBOR_USER = credentials('HARBOR_ACCOUNT')
}
steps {
# 指定使用docker 容器
container(name: 'docker') {
sh """
# 执行build 命令,Dockerfile 会在下一小节创建,也是放在代码仓库,和
Jenkinsfile 同级
docker build -t
${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
# 登录Harbor,HARBOR_USER_USR 和HARBOR_USER_PSW 由上述environment 生
成
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW}
${HARBOR_ADDRESS}
# 将镜像推送至镜像仓库
docker push
${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
(5).发版到kubernets集群
最后一步就是将该镜像发版至Kubernetes 集群中,此时使用的是包含kubectl 命令的容器:
stage('Deploying to K8s') {
# 获取连接Kubernetes 集群证书
environment {
MY_KUBECONFIG = credentials('study-k8s-kubeconfig')
}
steps {
# 指定使用kubectl 容器
container(name: 'kubectl'){
sh """
# 直接set 更改Deployment 的镜像即可
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image
deploy -l app=${IMAGE_NAME}
${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n
$NAMESPACE
"""
}
}
}
}
注意事项:
本次发版的命令为/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE。
由于本示例的Deployment 名称、Label、容器名称均和项目名称一致,所以就统一了IMAGE_NAME,如果示例不是按照此规范命名,需要进行更改。
4.定义Dockerfile
在执行流水线过程时,需要将代码的编译产物做成镜像。Dockerfile 主要写的是如何生成公司业务的镜像。而本次示例是Java 项目,只需要把Jar 包放在有Jre 环境的镜像中,然后启动该Jar 包即可:
gitlab→选择项目→新建文件:Dockerfile
# 基础镜像可以按需修改,可以更改为公司自有镜像 FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data # jar包名称改成实际的名称,本示例为spring-cloud-eureka-0.0.1-SNAPSHOT.jar COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./ # 启动Jar包 CMD java -jar spring-cloud-eureka-0.0.1-SNAPSHOT.jar
5.定义Kubernetes资源
本示例在 GitLab 创建的 Group 为 kubernetes,可以将其认为是一个项目,同一个项目可以部署至Kubernetes 集群中同一个Namespace 中,本示例为kubernetes 命名空间。由于使用的是私有仓库,因此也需要先配置拉取私有仓库镜像的密钥:
# kubectl create ns kubernetes namespace/kubernetes created # kubectl create secret docker-registry harborkey --docker-server=192.168.0.110 --docker-username=admin --docker-password=admin123 --docker-email=mail@kubeasy.com -n kubernetes secret/harborkey created
配置该应用的Deployment:
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: spring-boot-project
name: spring-boot-project
namespace: kubernetes
spec:
ports: # 端口按照实际情况进行修改
- name: web
port: 8761
protocol: TCP
targetPort: 8761
selector:
app: spring-boot-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: spring-boot-project
namespace: kubernetes
spec:
rules:
- host: spring-boot-project.test.com #设置对外访问域名,根据实际需求更改
http:
paths:
- backend:
service:
name: spring-boot-project
port:
number: 8761
path: /
pathType: ImplementationSpecific
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: spring-boot-project # Deployment 标签,和流水线的set -l 一致
name: spring-boot-project # Deployment 名称
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: spring-boot-project # Pod 的标签
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: spring-boot-project # Pod 的标签
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- spring-boot-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: nginx # 此处使用的nginx作为原始的镜像,通过Jenkins构建并发版后,变成Java应用的镜像
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8761 # 端口号和健康检查按照实际情况进行修改
timeoutSeconds: 2
name: spring-boot-project # 容器的名称,需要和流水线set 命令的容器名称一
ports:
- containerPort: 8761 # 端口号按照实际情况进行修改
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8761 # 端口号和健康检查按照实际情况进行修改
timeoutSeconds: 2
resources: # 资源请求按照实际情况修改
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey # Harbor仓库密钥,需要和上述创建的Secret 一致
restartPolicy: Always
securityContext: {}
serviceAccountName: default
创建该资源:
# kubectl create -f spring-boot-project.yaml service/spring-boot-project created ingress.networking.k8s.io/spring-boot-project created deployment.apps/spring-boot-project created
6.创建jenkins job任务
单击首页的创建任务(Job)选项(或者是New Item)并配置任务信息:
新建任务→选择流水线:
在新页面,点击Pipeline,输入Git 仓库地址、选择之前配置GitLab Key、分支为master,点击Save 即可:

创建完成后,点击 Build Now(由于 Jenkins 参数由 Jenkinsfile 生成,所以第一次执行流水线会失败):

选择分支,之后点击Build,然后点击Build History 的进度条即可看到构建日志:

构建日志上部分为创建Pod的日志,可以看到Pod为Agent指定的Pod:

可以看到BUILD SUCCESS 说明mvn 编译已经通过:

编译结束后,可以看到制作镜像的日志,并推送到仓库:

Finished:SUCCESS说明该流水线正常结束。接下来可以查看Deployment的镜像:
[root@k8s-master01 ~]# kubectl get deploy -n kubernetes spring-boot-project -oyaml | grep "image"
f:imagePullPolicy: {}
f:imagePullSecrets:
f:image: {}
image: 192.168.0.110/kubernetes/spring-boot-project:jenkins-spring-boot-project-7-039cf7e
imagePullPolicy: IfNotPresent
imagePullSecrets
如果配置了域名,可以通过域名访问(测试域名需要配置hosts):

二.自动化构建Vue/H5前端应用
自动化构建 Vue/H5 应用,其构建方式和自动化构建 Java 基本相同,重点是更改Deployment、Jenkinsfile 和Dockerfile 即可。
前端应用测试项目地址:https://gitee.com/dukuan/vue-project.git
1.定义Jenkinsfile
和上述java项目类似,只需要改部分内容即可:gitlab项目地址、nodejs环境镜像、部分标签。
pipeline {
agent {
kubernetes {
cloud 'study-kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/node:lts" # 构建容器改为具有NodeJS 环境的镜像,版本需要和公司项目一致
imagePullPolicy: "IfNotPresent"
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/root/.m2/" # NodeJS 的缓存目录为node_modules,此处可以
不配置,因为workspace 采用的是hostPath,该目录会被缓存到创建Pod 节点的/opt/目录
name: "cachedir"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: "IfNotPresent"
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: "IfNotPresent"
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
}
stages {
stage('Pulling Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git(changelog: true, poll: true, url: 'git@192.168.0.111:root/vue-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key') # 此处的地址需要改为自己的Git 地址
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git(url: 'git@192.168.0.111:root/vue-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key') # 此处的地址需要改为自己的Git 地址
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
}
}
stage('Building') {
steps {
container(name: 'build') {
sh """ # 此处为NodeJS/Vue 项目的构建命令,根据实际情况进行修改
npm install --registry=https://registry.npm.taobao.org
npm run build
"""
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('HARBOR_ACCOUNT')
}
steps {
container(name: 'docker') {
sh """
echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8s') {
environment {
MY_KUBECONFIG = credentials('study-kubernetes')
}
steps {
container(name: 'kubectl'){
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.0.110"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "vue-project" # 项目标签名字为vue-project
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH')
}
}
2.定义Dockerfile
FROM registry.cn-beijing.aliyuncs.com/dotbalo/nginx:1.15.12 COPY dist/* /usr/share/nginx/html/
3.定义Kubernetes资源
对于Kubernetes 的资源也是类似的,只需要更改资源名称和端口号即可:
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: vue-project
name: vue-project
namespace: kubernetes
spec:
ports:
- name: web
port: 80
protocol: TCP
targetPort: 80
selector:
app: vue-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: vue-project
namespace: kubernetes
spec:
rules:
- host: vue-project.test.com
http:
paths:
- backend:
service:
name: vue-project
port:
number: 80
path: /
pathType: ImplementationSpecific
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vue-project
name: vue-project
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: vue-project
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: vue-project
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- vue-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: nginx
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
name: vue-project
ports:
- containerPort: 80
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
resources:
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey
restartPolicy: Always
securityContext: {}
serviceAccountName: default
创建该资源:
# kubectl create -f vue-project.yaml service/vue-project created ingress.networking.k8s.io/vue-project created deployment.apps/vue-project created
4.创建Jenkins Job
创建Jenkins Job 和之前并无太大区别。首先创建Pipeline 类型的Job,名称为vue-project:


三.自动化构建Golang项目
上述演示了Java 和前端应用的自动化,接下来演示一下对于Golang 的自动化构建,本次示例的代码地址:https://gitee.com/dukuan/go-project.git。
1.定义Jenkinsfile
本次示例的Jenkinsfile 和之前的也无太大区别,需要的更改的位置是构建容器的镜像、缓存目录、Git 地址和项目名称
pipeline {
agent {
kubernetes {
cloud 'study-kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/golang:1.15" # 构建镜像改为golang,需要根据实际情况更改版本
imagePullPolicy: "IfNotPresent"
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/go/pkg" #增加挂载缓存目录
name: "cachedir"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: "IfNotPresent"
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: "IfNotPresent"
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir" # 缓存目录保存的位置
hostPath:
path: "/opt/gopkg"
'''
}
}
stages {
stage('Pulling Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git(changelog: true, poll: true, url: 'git@192.168.0.111:root/go-project.git', branch: "${BRANCH}", credentialsId: 'gitlab key')
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git(url: 'git@192.168.0.111:root/go-project.git', branch: env.gitlabBranch, changelog: true,poll: true, credentialsId: 'gitlab key')
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
}
}
stage('Building') {
steps {
container(name: 'build') {
sh """ # 构建命令
export GO111MODULE=on
go env -w GOPROXY=https://goproxy.cn,direct
go build
"""
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('HARBOR_ACCOUNT')
}
steps {
container(name: 'docker') {
sh """
echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8s') {
environment {
MY_KUBECONFIG = credentials('study-kubernetes')
}
steps {
container(name: 'kubectl'){
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.0.110"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "go-project"
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*',type: 'PT_BRANCH')
}
}
2.定义Dockerfile
FROM registry.cn-beijing.aliyuncs.com/dotbalo/alpine-glibc:alpine-3.9 # 如果定义了单独的配置文件,可能需要拷贝到镜像中 COPY conf/ ./conf # 包名按照实际情况进行修改 COPY ./go-project ./ ENTRYPOINT [ "./go-project"] # 启动该应用
3.定义Kubernetes资源
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: go-project
name: go-project
namespace: kubernetes
spec:
ports:
- name: web
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: go-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: go-project
namespace: kubernetes
spec:
rules:
- host: go-project.test.com
http:
paths:
- backend:
service:
name: go-project
port:
number: 8080
path: /
pathType: ImplementationSpecific
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: go-project
name: go-project
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: go-project
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: go-project
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- go-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: nginx
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 2
name: go-project
ports:
- containerPort: 8080
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 2
resources:
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey
restartPolicy: Always
securityContext: {}
serviceAccountName: default
4.创建Jenkins Job
直接创建即可:
# kubectl create -f go-project.yaml service/go-project created ingress.networking.k8s.io/go-project created deployment.apps/go-project created
构建省略:

四.自动触发构建
之前的构建都是采用手动选择分支进行构建的,实际使用时,项目可能有很多,如果都是手动触发可能比较消耗人力。所以推荐可以按需配置自动触发,即提交代码后自动触发Jenkins 进行构建任务。
本次用Java 项目进行演示。首先找到Java 项目的Job,点击Configure:
之后选择构建触发器(Build Triggers),勾选Build when a change…,记录webhook URL:

之后点击高级(Advanced):

选择Allow all branches,如果不想任何分支都可以触发该流水线,可以选择Filter 进行条件匹配。之后点击Generate 生成Secret token,最后点击Save 即可。
接下来配置GitLab,首先点击Menu→Admin,选择Settings→Network,之后允许访问外部的请求:

保存后,找到Java 项目,点击Settings→WebHooks,在新页面输入Webhook 地址和token:

之后下方会添加一个新的Project Hooks,可以点击Test 进行Push 测试:

点击后,即可在Jenkins 页面看到任务被触发:

五.UAT和生产环境发版
UAT和生产环境最好跟测试环境分开,使用测试构建好的镜像经行发版:
1.如果生产环境用同样构建方式,那么生产构建的新镜像有可能和上次的不一样,新镜像没有经过测试。
2.使用测试环境测试过的镜像可以省去编译构建的时间。
流程简略:
创建一个新的Job,名字为go-project-uat,类型Pipeline:点击页面的This Project is parameterized(参数化构建):
选择参数类型为Image Tag Parameter(需要安装Image Tag 插件),之后定义Name 为变量的名称,Iamge Name 为Harbor 的目录和镜像名称:

点击Advance,输入仓库的地址,注意如果配置了证书,需要配置https:

之后可以先点击Save,然后测试能否获取到进行Tag。保存后点击Build with Parameters

之后可以直接添加Pipeline脚本,也可以在gitlab中用jenkinsfile来创建:

完整代码:
pipeline {
agent {
kubernetes {
cloud 'study-kubernetes'
slaveConnectTimeout 1200
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
name: jnlp
imagePullPolicy: IfNotPresent
- command:
- "cat"
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: "IfNotPresent"
name: "kubectl"
tty: true
restartPolicy: "Never"
'''
}
}
stages {
stage('Deploy') {
environment {
MY_KUBECONFIG = credentials('study-kubernetes')
}
steps {
container(name: 'kubectl'){
sh """
echo ${IMAGE_TAG}
kubectl --kubeconfig=${MY_KUBECONFIG} set image deployment -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${IMAGE_TAG} -n ${NAMESPACE}
kubectl --kubeconfig=${MY_KUBECONFIG} get po -l app=${IMAGE_NAME} -n ${NAMESPACE} -w
"""
}
}
}
}
environment {
HARBOR_ADDRESS = "HARBOR_ADDRESS"
NAMESPACE = "kubernetes"
IMAGE_NAME = "spring-boot-project"
TAG = ""
}
}
保存后,选择一个镜像,点击Build,即可看到是直接将镜像版本更新至Kubernetes,并无构建过程,可以省下很多时间。该流水
线也可以选择之前的版本进行回滚操作。

