kubernetes Jenkins gitlab搭建CI/CD环境 (二)
May242018
接前一篇文章 kubernetes Jenkins gitlab搭建CI/CD环境–(一)，本文介绍在Kubernetes上安装Jenkins。
Jenkins的安装有多种，可以在独立的服务器安装，结合K8S的话可以使用helm，参考：https://github.com/kubernetes/charts/tree/master/stable/jenkins
chart中使用的Jenkins基础镜像 jenkins/jenkins:lts，也可以通过Dockerfile自己定制。
下面是我基于jenkins/jenkins:lts定制的Dockerfile，增加了docker，docker-compose，kubectl和maven，然后使用yaml文件手动部署的。
Jenkins Master的 Dockerfile 如下：
FROM jenkins/jenkins:lts
MAINTAINER Fisher.yu <yu2hei@gmail.com>
EXPOSE 8080 50000
ENV DOCKER_VERSION=17.04.0-ce DOCKER_COMPOSE_VERSION=1.21.2 KUBECTL_VERSION=v1.10.1
# Use Root to setup kubectl, docker-ce, docker-compose
USER root
WORKDIR /usr/local/bin
############
# change debian source to 163
############
RUN sed -i 's/deb.debian.org/mirrors.163.com/g' /etc/apt/sources.list
RUN sed -i 's/security.debian.org/mirrors.163.com\/debian-security/g' /etc/apt/sources.list
############
# Update packages
############
RUN apt-get update && apt-get install -y --no-install-recommends \
make \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
#######
# docker-ce
#######
#RUN curl -fsSLO https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz \
RUN curl -fsSLO http://sudops.com/docker-${DOCKER_VERSION}.tgz \
&& tar --strip-components=1 -xvzf docker-${DOCKER_VERSION}.tgz -C /usr/local/bin \
&& chmod -R +x /usr/local/bin/docker
#######
# docker-compose
#######
#RUN curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \
RUN curl -L http://sudops.com/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
#######
# kubectl
#######
RUN curl -L http://sudops.com/kubectl-${KUBECTL_VERSION} -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
#######
# Maven
#######
# Preparation
ENV MAVEN_VERSION 3.5.3
ENV MAVEN_HOME /etc/maven-${MAVEN_VERSION}
# Installation
RUN cd /tmp
RUN wget http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz
RUN mkdir maven-${MAVEN_VERSION}
RUN tar -zxvf apache-maven-${MAVEN_VERSION}-bin.tar.gz --directory maven-${MAVEN_VERSION} --strip-components=1
RUN mv maven-${MAVEN_VERSION} ${MAVEN_HOME}
ENV PATH ${PATH}:${MAVEN_HOME}/bin
# Cleanup
RUN rm apache-maven-${MAVEN_VERSION}-bin.tar.gz
RUN unset MAVEN_VERSION
#######
# Back to Jenkins home
#######
USER jenkins
WORKDIR $JENKINS_HOME
#build docker 镜像
docker build -t repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1 --no-cache .
#将build好的docker镜像push到私有docker-harbor中
docker push repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1
创建Jenkins部署文件
*** 本文中kubernetes使用namespace均为devns
创建jenkins PersistentVolumeClaim yaml文件
# cat jenkins-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jenkins-pvc
namespace: devns
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
storageClassName: kyglustersc
** 说明，这里指定了之前创建的storageclass： kyglustersc
创建jenkins deployment文件
# cat jenkins-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kyjenkins
namespace: devns
labels:
app: kyjenkins
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: kyjenkins
tier: kyjenkins
spec:
containers:
- image: repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1
name: kyjenkins
securityContext:
privileged: true
ports:
- containerPort: 8080
name: kyjenkins
- containerPort: 50000
name: agent
protocol: TCP
volumeMounts:
- name: docker
mountPath: /var/run/docker.sock
- name: jenkins-persistent-storage
mountPath: /var/jenkins_home
- name: kube-config
mountPath: /root/.kube/config
volumes:
- name: docker
hostPath:
path: /var/run/docker.sock
- name: jenkins-persistent-storage
persistentVolumeClaim:
claimName: jenkins-pvc
- name: kube-config
hostPath:
path: /root/.kube/config
创建jenkins-service
# cat jenkins-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kyjenkins
namespace: devns
labels:
app: kyjenkins
spec:
ports:
- port: 8080
targetPort: 8080
name: kyjenkins
- port: 50000
targetPort: 50000
name: agent
selector:
app: kyjenkins
tier: kyjenkins
然后执行
kubectl create -f .
待jenkisn pods running后，describe查看pods情况：
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2d (x5 over 2d) default-scheduler pod has unbound PersistentVolumeClaims (repeated 9 times) Normal Scheduled 2d default-scheduler Successfully assigned kyjenkins-8685458884-j8qjg to node_178 Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "docker" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "kube-config" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "default-token-fxgsp" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "pvc-d7ad4f36-5d73-11e8-bcc6-5254006a334a" Normal Pulled 2d kubelet, node_178 Container image "repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1" already present on machine Normal Created 2d kubelet, node_178 Created container Normal Started 2d kubelet, node_178 Started container
Jenkins Master运行没问题，然后配置Ingress，并reload配置。
- host: jenkins.kydev.in
http:
paths:
- backend:
serviceName: kyjenkins
servicePort: 8080
访问 http://jenkins.kydev.in
解锁密钥可以在kubernetes上执行 kubectl logs kyjenkins-8685458884-j8qjg -f 查看，也可以exec 进入到pods内部查看文件。
可以安装下社区推荐的插件：
Jenkins一些常用的插件，可以根据自己的实际情况安装：
locale php-jenkins-plugins checkstyle cloverphp dry htmlpublisher jdepend plot pmd violations xunit php git phing build-pipeline-plugin bouncycastle API PHP Built-in Web Server ElasticBox Jenkins Kubernetes CI/CD Javadoc Maven Integration OWASP Markup Formatter Static Analysis Utilities DRY GitLab Logo Kubernetes :: Pipeline :: Arquillian Steps Gitlab Merge Request Builder Kubernetes Cli Kubernetes :: Pipeline :: DevOps Steps Kubernetes :: Pipeline :: Kubernetes Steps Kubernetes :: Pipeline :: Aggregator Windows Slaves Matrix Authorization Strategy Phing Plot Gitlab Authentication Violation Comments to GitLab Token Macro Ant PAM Authentication LDAP External Monitor Job Type Run Condition Conditional BuildStep Parameterized Trigger jQuery Build Pipeline ruby-runtime Gitlab Hook xUnit Pipeline Aggregator GitHub API GitHub GitHub Branch Source Pipeline: GitHub Clover PHP JDepend Dashboard View Delivery Pipeline HTML Publisher PMD Checkstyle Violations php Checkstyle Azure Commons Kubernetes Continuous Deploy Kubernetes Credentials Provider GitLab Kubernetes :: Pipeline :: Kubernetes Steps Kubernetes :: Pipeline :: DevOps Steps
如果使用默认源比较慢，可以多试试其他几个源：
https://updates.jenkins.io/update-center.json (默认：) http://updates.jenkins-ci.org/update-center.json https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json http://mirror.xmission.com/jenkins/updates/current/update-center.json
系统管理–云–增加Kubernets连接配置
接下来可以验证Master的工作情况：
创建一个freestyle的job kube-test。
可以看到任务执行成功，kubectl get pods -n devns 可以打印出结果。
接下来配置Jenkins slave
同样Jenkins slave也是采用自定义Dockerfile的方式
# cat Dockerfile
FROM jenkins/jnlp-slave
MAINTAINER Fisher.yu <yu2hei@gmail.com>
ENV KUBECTL_VERSION=v1.10.1 DOCKER_VERSION=17.04.0-ce
USER root
##########
# Maven-3.5.3
##########
COPY maven /usr/share/maven/
RUN chmod +x /usr/share/maven/bin/mvn && ln -s /usr/share/maven/bin/mvn /usr/local/bin/mvn
##########
# kubectl-1.10.1
##########
COPY kubectl /usr/local/bin/
RUN chmod +x /usr/local/bin/kubectl
##########
# 预配置 kubectl
##########
##########
# 在运行时由 ConfigMap 挂载Volume
##########
ENV CERT_DIR /etc/kubernetes/conf
ARG DOCKER_SOCK_DIR=/var/run/docker.sock
RUN mkdir -p ${CERT_DIR} \
mkidr -p /root/.kube \
touch ${CERT_DIR}/k8s-root-ca.pem \
touch ${CERT_DIR}/admin.pem \
touch ${CERT_DIR}/admin-key.pem
COPY config /usr/local/bin/
RUN export KUBECONFIG=/usr/local/bin/config && kubectl config view
##########
# docker-ce
##########
#------------------------------------------------#
copy docker /usr/bin/docker
RUN apt-get -y update && apt-get install -y apt-utils iptables libdevmapper1.02.1 libltdl7 libseccomp2 && apt-get -y autoremove && chmod +x /usr/bin/docker
##########
# 挂载volume
##########
VOLUME ${CERT_DIR}
VOLUME ${DOCKER_SOCK_DIR}
## 当前目录结果
# tree .
.
├── config
├── docker
├── docker-compose
├── Dockerfile
├── kubectl
└── maven
├── bin
├── conf
├── lib
└── README.txt
# cat config
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/conf/k8s-root-ca.pem
server: http://k8sapi.kydev.in
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: kube-system
user: admin
name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate: /etc/kubernetes/conf/admin.pem
client-key: /etc/kubernetes/conf/admin-key.pem
将build的镜像push到私有repo中。
docker build -t repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1 --no-cache . docker push repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1
# (1). 创建 kubectl-cert-cm 资源对象
kubectl create configmap kubectl-cert-cm --from-file=/etc/kubernetes/ssl -n=devns
# (2). 查看所创建的 kubectl-cert-cm 资源对象
kubectl describe configmap kubectl-cert-cm -n=devns
下面继续配置Jenkins的Slave节点。
Kubernetes Pod Template 中配置，详见截图
好了，现在Jenkins的jnlp-slave已经配置完毕
下面开始测试：
创建pipeline任务
pipeline script
def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, containers: [
containerTemplate(name: 'maven', image: 'repo.ky.in/webcola/jdk-maven-ant:v1.0.0', ttyEnabled: true, command: 'cat')
]) {
node(label) {
stage('Get a Maven project') {
container('maven') {
stage('wait for exec check'){
sh 'sleep 10'
}
stage('get maven env') {
sh 'cat /etc/resolv.conf'
sh 'cat /etc/issue'
sh 'uname -a'
sh 'env'
sh 'echo "$(sed \'s/options ndots:5/#options ndots:5/g\' /etc/resolv.conf)" > /etc/resolv.conf'
}
stage('code checkout') {
git url: 'http://gitlab.ky.in/devops/tomcatwartest.git', credentialsId: '29efa3cc-4fe7-42e2-95a4-a03c18d56603', branch: 'master'
sh 'mvn clean package'
sh 'sleep 300'
}
}
}
}
}
任务运行成功。
在配置Jenkins-slave是有几个地方需要注意：
（1）Jenkins slave使用自定义镜像调用kubectl时freestyle的任务可以在构建环境中指定『setup kubernetes CLI』，但是在pipeline中就需要在镜像里面做文章了，可以参考jenkins-slave的Dockerfile，需要将三个证书文件和config文件传入到镜像中，创建kubectl-cert-cm的configmap。
（2）pipeline中的podTemplate指定了自定义的image，但是实际启动中还是会默认再启动一个jenkins/jnlp-slave:alpine的镜像。然而这个alpine本身就有很多”坑”，比如dns解析的问题，在slave的镜像里面git代码会报Can’t resolve git server host。解决方式就是注释掉/etc/resolv.conf里面的”options ndots:5″ 这行，具体见我之前的文章：kubernetes 使用基于 alpine 镜像无法正常解析外网DNS
好了，接下来的文章会介绍kubernetes上安装gitlab，配置钩子自动出发流程，以及本文涉及的一个简单的tomcat-maven测试环境。
本文固定链接: https://sudops.com/kubernetes-jenkins-gitlab-ci-cd-env-2.html | 运维速度
【上一篇】kubernetes Jenkins gitlab搭建CI/CD环境–(一)
【下一篇】解决 AttributeError: ‘module’ object has no attribute ‘SSL_ST_INIT’
【下一篇】解决 AttributeError: ‘module’ object has no attribute ‘SSL_ST_INIT’
最活跃的读者