kubernetes Jenkins gitlab搭建CI/CD环境 (二)
May242018
接前一篇文章 kubernetes Jenkins gitlab搭建CI/CD环境–(一)，本文介绍在Kubernetes上安装Jenkins。
Jenkins的安装有多种，可以在独立的服务器安装，结合K8S的话可以使用helm，参考：https://github.com/kubernetes/charts/tree/master/stable/jenkins
chart中使用的Jenkins基础镜像 jenkins/jenkins:lts，也可以通过Dockerfile自己定制。
下面是我基于jenkins/jenkins:lts定制的Dockerfile，增加了docker，docker-compose，kubectl和maven，然后使用yaml文件手动部署的。
Jenkins Master的 Dockerfile 如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
FROM jenkins/jenkins:lts MAINTAINER Fisher.yu <yu2hei@gmail.com> EXPOSE 8080 50000 ENV DOCKER_VERSION=17.04.0-ce DOCKER_COMPOSE_VERSION=1.21.2 KUBECTL_VERSION=v1.10.1 # Use Root to setup kubectl, docker-ce, docker-compose USER root WORKDIR /usr/local/bin ############ # change debian source to 163 ############ RUN sed -i 's/deb.debian.org/mirrors.163.com/g' /etc/apt/sources.list RUN sed -i 's/security.debian.org/mirrors.163.com\/debian-security/g' /etc/apt/sources.list ############ # Update packages ############ RUN apt-get update && apt-get install -y --no-install-recommends \ make \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ####### # docker-ce ####### #RUN curl -fsSLO https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz \ RUN curl -fsSLO http://sudops.com/docker-${DOCKER_VERSION}.tgz \ && tar --strip-components=1 -xvzf docker-${DOCKER_VERSION}.tgz -C /usr/local/bin \ && chmod -R +x /usr/local/bin/docker ####### # docker-compose ####### #RUN curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \ RUN curl -L http://sudops.com/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \ && chmod +x /usr/local/bin/docker-compose ####### # kubectl ####### RUN curl -L http://sudops.com/kubectl-${KUBECTL_VERSION} -o /usr/local/bin/kubectl \ && chmod +x /usr/local/bin/kubectl ####### # Maven ####### # Preparation ENV MAVEN_VERSION 3.5.3 ENV MAVEN_HOME /etc/maven-${MAVEN_VERSION} # Installation RUN cd /tmp RUN wget http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz RUN mkdir maven-${MAVEN_VERSION} RUN tar -zxvf apache-maven-${MAVEN_VERSION}-bin.tar.gz --directory maven-${MAVEN_VERSION} --strip-components=1 RUN mv maven-${MAVEN_VERSION} ${MAVEN_HOME} ENV PATH ${PATH}:${MAVEN_HOME}/bin # Cleanup RUN rm apache-maven-${MAVEN_VERSION}-bin.tar.gz RUN unset MAVEN_VERSION ####### # Back to Jenkins home ####### USER jenkins WORKDIR $JENKINS_HOME |
#build docker 镜像
|
1 |
docker build -t repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1 --no-cache . |
#将build好的docker镜像push到私有docker-harbor中
|
1 |
docker push repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1 |
创建Jenkins部署文件
*** 本文中kubernetes使用namespace均为devns
创建jenkins PersistentVolumeClaim yaml文件
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# cat jenkins-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: jenkins-pvc namespace: devns spec: accessModes: - ReadWriteOnce resources: requests: storage: 60Gi storageClassName: kyglustersc ** 说明，这里指定了之前创建的storageclass： kyglustersc |
创建jenkins deployment文件
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# cat jenkins-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kyjenkins namespace: devns labels: app: kyjenkins spec: strategy: type: Recreate template: metadata: labels: app: kyjenkins tier: kyjenkins spec: containers: - image: repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1 name: kyjenkins securityContext: privileged: true ports: - containerPort: 8080 name: kyjenkins - containerPort: 50000 name: agent protocol: TCP volumeMounts: - name: docker mountPath: /var/run/docker.sock - name: jenkins-persistent-storage mountPath: /var/jenkins_home - name: kube-config mountPath: /root/.kube/config volumes: - name: docker hostPath: path: /var/run/docker.sock - name: jenkins-persistent-storage persistentVolumeClaim: claimName: jenkins-pvc - name: kube-config hostPath: path: /root/.kube/config |
创建jenkins-service
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# cat jenkins-service.yaml apiVersion: v1 kind: Service metadata: name: kyjenkins namespace: devns labels: app: kyjenkins spec: ports: - port: 8080 targetPort: 8080 name: kyjenkins - port: 50000 targetPort: 50000 name: agent selector: app: kyjenkins tier: kyjenkins |
然后执行
|
1 |
kubectl create -f . |
待jenkisn pods running后，describe查看pods情况：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2d (x5 over 2d) default-scheduler pod has unbound PersistentVolumeClaims (repeated 9 times) Normal Scheduled 2d default-scheduler Successfully assigned kyjenkins-8685458884-j8qjg to node_178 Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "docker" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "kube-config" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "default-token-fxgsp" Normal SuccessfulMountVolume 2d kubelet, node_178 MountVolume.SetUp succeeded for volume "pvc-d7ad4f36-5d73-11e8-bcc6-5254006a334a" Normal Pulled 2d kubelet, node_178 Container image "repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1" already present on machine Normal Created 2d kubelet, node_178 Created container Normal Started 2d kubelet, node_178 Started container |
Jenkins Master运行没问题，然后配置Ingress，并reload配置。
|
1 2 3 4 5 6 |
- host: jenkins.kydev.in http: paths: - backend: serviceName: kyjenkins servicePort: 8080 |
访问 http://jenkins.kydev.in
解锁密钥可以在kubernetes上执行 kubectl logs kyjenkins-8685458884-j8qjg -f 查看，也可以exec 进入到pods内部查看文件。
可以安装下社区推荐的插件：
Jenkins一些常用的插件，可以根据自己的实际情况安装：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
locale php-jenkins-plugins checkstyle cloverphp dry htmlpublisher jdepend plot pmd violations xunit php git phing build-pipeline-plugin bouncycastle API PHP Built-in Web Server ElasticBox Jenkins Kubernetes CI/CD Javadoc Maven Integration OWASP Markup Formatter Static Analysis Utilities DRY GitLab Logo Kubernetes :: Pipeline :: Arquillian Steps Gitlab Merge Request Builder Kubernetes Cli Kubernetes :: Pipeline :: DevOps Steps Kubernetes :: Pipeline :: Kubernetes Steps Kubernetes :: Pipeline :: Aggregator Windows Slaves Matrix Authorization Strategy Phing Plot Gitlab Authentication Violation Comments to GitLab Token Macro Ant PAM Authentication LDAP External Monitor Job Type Run Condition Conditional BuildStep Parameterized Trigger jQuery Build Pipeline ruby-runtime Gitlab Hook xUnit Pipeline Aggregator GitHub API GitHub GitHub Branch Source Pipeline: GitHub Clover PHP JDepend Dashboard View Delivery Pipeline HTML Publisher PMD Checkstyle Violations php Checkstyle Azure Commons Kubernetes Continuous Deploy Kubernetes Credentials Provider GitLab Kubernetes :: Pipeline :: Kubernetes Steps Kubernetes :: Pipeline :: DevOps Steps |
如果使用默认源比较慢，可以多试试其他几个源：
|
1 2 3 4 |
https://updates.jenkins.io/update-center.json (默认：) http://updates.jenkins-ci.org/update-center.json https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json http://mirror.xmission.com/jenkins/updates/current/update-center.json |
系统管理–云–增加Kubernets连接配置
接下来可以验证Master的工作情况：
创建一个freestyle的job kube-test。
可以看到任务执行成功，kubectl get pods -n devns 可以打印出结果。
接下来配置Jenkins slave
同样Jenkins slave也是采用自定义Dockerfile的方式
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
# cat Dockerfile FROM jenkins/jnlp-slave MAINTAINER Fisher.yu <yu2hei@gmail.com> ENV KUBECTL_VERSION=v1.10.1 DOCKER_VERSION=17.04.0-ce USER root ########## # Maven-3.5.3 ########## COPY maven /usr/share/maven/ RUN chmod +x /usr/share/maven/bin/mvn && ln -s /usr/share/maven/bin/mvn /usr/local/bin/mvn ########## # kubectl-1.10.1 ########## COPY kubectl /usr/local/bin/ RUN chmod +x /usr/local/bin/kubectl ########## # 预配置 kubectl ########## ########## # 在运行时由 ConfigMap 挂载Volume ########## ENV CERT_DIR /etc/kubernetes/conf ARG DOCKER_SOCK_DIR=/var/run/docker.sock RUN mkdir -p ${CERT_DIR} \ mkidr -p /root/.kube \ touch ${CERT_DIR}/k8s-root-ca.pem \ touch ${CERT_DIR}/admin.pem \ touch ${CERT_DIR}/admin-key.pem COPY config /usr/local/bin/ RUN export KUBECONFIG=/usr/local/bin/config && kubectl config view ########## # docker-ce ########## #------------------------------------------------# copy docker /usr/bin/docker RUN apt-get -y update && apt-get install -y apt-utils iptables libdevmapper1.02.1 libltdl7 libseccomp2 && apt-get -y autoremove && chmod +x /usr/bin/docker ########## # 挂载volume ########## VOLUME ${CERT_DIR} VOLUME ${DOCKER_SOCK_DIR} |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## 当前目录结果 # tree . . ├── config ├── docker ├── docker-compose ├── Dockerfile ├── kubectl └── maven ├── bin ├── conf ├── lib └── README.txt |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# cat config apiVersion: v1 clusters: - cluster: certificate-authority: /etc/kubernetes/conf/k8s-root-ca.pem server: http://k8sapi.kydev.in name: kubernetes contexts: - context: cluster: kubernetes namespace: kube-system user: admin name: kubernetes current-context: kubernetes kind: Config preferences: {} users: - name: admin user: client-certificate: /etc/kubernetes/conf/admin.pem client-key: /etc/kubernetes/conf/admin-key.pem |
将build的镜像push到私有repo中。
|
1 2 |
docker build -t repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1 --no-cache . docker push repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1 |
|
1 2 3 4 5 |
# (1). 创建 kubectl-cert-cm 资源对象 kubectl create configmap kubectl-cert-cm --from-file=/etc/kubernetes/ssl -n=devns # (2). 查看所创建的 kubectl-cert-cm 资源对象 kubectl describe configmap kubectl-cert-cm -n=devns |
下面继续配置Jenkins的Slave节点。
Kubernetes Pod Template 中配置，详见截图
好了，现在Jenkins的jnlp-slave已经配置完毕
下面开始测试：
创建pipeline任务
pipeline script
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
def label = "mypod-${UUID.randomUUID().toString()}" podTemplate(label: label, containers: [ containerTemplate(name: 'maven', image: 'repo.ky.in/webcola/jdk-maven-ant:v1.0.0', ttyEnabled: true, command: 'cat') ]) { node(label) { stage('Get a Maven project') { container('maven') { stage('wait for exec check'){ sh 'sleep 10' } stage('get maven env') { sh 'cat /etc/resolv.conf' sh 'cat /etc/issue' sh 'uname -a' sh 'env' sh 'echo "$(sed \'s/options ndots:5/#options ndots:5/g\' /etc/resolv.conf)" > /etc/resolv.conf' } stage('code checkout') { git url: 'http://gitlab.ky.in/devops/tomcatwartest.git', credentialsId: '29efa3cc-4fe7-42e2-95a4-a03c18d56603', branch: 'master' sh 'mvn clean package' sh 'sleep 300' } } } } } |
任务运行成功。
在配置Jenkins-slave是有几个地方需要注意：
（1）Jenkins slave使用自定义镜像调用kubectl时freestyle的任务可以在构建环境中指定『setup kubernetes CLI』，但是在pipeline中就需要在镜像里面做文章了，可以参考jenkins-slave的Dockerfile，需要将三个证书文件和config文件传入到镜像中，创建kubectl-cert-cm的configmap。
（2）pipeline中的podTemplate指定了自定义的image，但是实际启动中还是会默认再启动一个jenkins/jnlp-slave:alpine的镜像。然而这个alpine本身就有很多”坑”，比如dns解析的问题，在slave的镜像里面git代码会报Can’t resolve git server host。解决方式就是注释掉/etc/resolv.conf里面的”options ndots:5″ 这行，具体见我之前的文章：kubernetes 使用基于 alpine 镜像无法正常解析外网DNS
好了，接下来的文章会介绍kubernetes上安装gitlab，配置钩子自动出发流程，以及本文涉及的一个简单的tomcat-maven测试环境。
本文固定链接: https://www.sudops.com/kubernetes-jenkins-gitlab-ci-cd-env-2.html | 运维·速度
【上一篇】kubernetes Jenkins gitlab搭建CI/CD环境–(一)
【下一篇】解决 AttributeError: ‘module’ object has no attribute ‘SSL_ST_INIT’
【下一篇】解决 AttributeError: ‘module’ object has no attribute ‘SSL_ST_INIT’
kubernetes Jenkins gitlab搭建CI/CD环境 (二)：等您坐沙发呢！
发表评论
欢迎关注 运维·速度 微信公众号
