Simpler chronicle of CI(Continuous Integration) “乱弹系列”之持续集成工具

引言

有句话说有人的地方就有江湖,同样,有江湖的地方就有恩怨。在软件行业历史长河(虽然相对于其他行业来说,软件行业的历史实在太短了,但是确是充满了智慧的碰撞也是十分的精彩)中有一些恩怨情愁,分分合合的小故事,比如类似的有,从一套代码发展出来后面由于合同到期就分道扬镳,然后各自发展成独门产品的Sybase DB和微软的SQL Server;另外一个例子是,当时JBPM的两个主要开发的小伙伴离开当时的RedHat,在JBPM基础上自立门户新创建的Java工作流管理软件Activiti,等等。在持续集成工具龙头老大这个宝座,也曾经发生过合作合并,吵架分家,再对着干的事情,今天分享一下这前前后后有趣的故事。

DevOps

首先,防止__先入为主__,以为大家都知道这个那个的。先普及下相关背景知识,如果已经了解的同学可以跳过。目前在软件工程领域已经火了好几年的DevOps领域,核心的模块就是CI与’CD’,即Continuous Integration与Continuous Deployment,也就是持续集成与持续部署,这个对于处于敏捷开发环境下尤其是互联网等需要高速迭代是个核心的功能,可以说没有CI,就不可能达到像Google或者Facebook这些一天有多个release的情况。

CI

CI(Continuous Integration) 持续集成起源于 XP(极限编程)与 TDD (Test Driven Develop)也就是__以测试驱动__的开发模式,是防止出现所谓的’集成地狱’,即防止程序员在正常编码工作中,需要写新的业务逻辑,添加新的代码,但是同时也新引入了bug。CI会持续的(重复的)进行一些小的工作,比如不断的跑测试用例,去扫描代码等工作。以减轻或者很大程度上避免这个个新引入的bug对软件交付质量引起的负面影响。目前,市场上有很多的CI解决方案及工具,常用的如下几个,

CI 的进化史

世界上本来没有CI,用的人多了也就成就了CI。本来软件工程里是没有这个概念的。最开始,就像下图中描述的帝国时代里,整个社会节奏平稳而缓慢,每个程序员自己做自己的开发,然后各自把自己的工作上次(提交),整个团队把代码放在一起,然后整个人过来,启动make/build,后面有个人去把编译好的代码放到测试机器上,每个程序员自己或者单独的测试团队去测试程序,如果没有问题,另外的人去发布到生产环境上。这些都是或多或少由人手工去做的。

但是就像很多人类的发明就是为了人类”偷懒”一样,CI慢慢在一些想偷懒的牛人脑子里形成。这其中就有Kent Beck (多说一句,这个现在工作于Facebook的牛人,还发明创造了很多到现在还在流行的东西,比如Agile敏捷开发,以JUnit为代码的xUnit测试理念,TDD测试驱动开发等等),在上个世纪最后几年,Kent Beck创造了XP(注意这个不是Bill的那个XP操作系统),是eXtreme Programming,即极限编程。虽然现在看起来极限编程有很多很诡异不太现实的方式,比如两个程序员坐在一起,使用一台电脑一起写一段程序等天马行空的想法。但是其中一个理念就是“持续集成”(CI)。以此理念,后面出现了使用各种语言写的CI的工具,其中的老大是CruiseControl。这个就像是上图中那个跑车一样,在当时整个缓慢的大环境下其提升工作效率的效果十分的吸眼。

到了2005年,当时就职于Sun(没错,就是创造了Java的那家公司)的一个叫川口浩介(Kohsuke Kawaguchi)的日本人,就是上图这位“霓虹金”,敢于冒险,重新“发明轮子”,不顾如日中天的CruiseControl,设计并开发了一个新的持续集成的软件,起名叫做Hudson。它提供了很多强大的功能,比如提供插件机制,这样就使其几乎集成了市面上所有的源代码管理工具,比如CVS, Subversion, Git, Perforce等。除此之外,它还提供了界面的扩展能力,另外还支持基于Apache Ant 和 Apache Maven的项目,除了xNix,还支持Windows环境等一众强大功能。听起来这么牛逼的工具,很快,在大约2007年的时候Hudson已经超越CruiseControl。然后在2008年5月的JavaOne大会上,Hudson获得了开发解决方案类的Duke’s Choice奖项。从此,小弟翻身做大哥,Hudson成为CI的代名词。其主要开发者 Kohsuke Kawaguchi 还获得了Google-O’Reilly Open Source Award。他后来也不用自己苦逼的写代码了,只要到处受邀去演讲做是如何受什么启发创造并发明了这么好的工具,造福大批程序员。再后来他还离职创立了公司CloudBees,出任CEO,迎娶白富美,走上人生新巅峰。(也难怪上图中他笑的如此开心)

一切看起来都是那么美好。但是,天有不测风云,在2009年6月,Oracle收购Sun,所有人都蒙逼了,是不是写反了?一个传统数据库的公司收购了在Java及开源老大的Sun?!!这个消息公布之后,两个公司内部各个产品及项目就被整合,调整,Hudson也不例外。这也就算了,反正谁给钱不是干活哪,但是在2010年9月,Oracle竟然暗戳啜的把Hudson变成了注册商标。2010年11月,Hudson社区的核心开发人员发现了这个事情,他们觉得这对于一个一直标榜自己是开源CI领域“诚实可靠小郎君”的Hudson来说是个玷污。双方进行了会谈,过程不太友好,然后就不出意料的谈崩了。2011年圣诞节过后,几个秃顶的大叔觉得不要再跟Oracle的律师在这里瞎扯淡了,他们决定自立门户,自己起个新的名字叫Jenkins。然后凑钱注册网址,买服务器,列出下面的清单,统统改名,

  • hudson-labs.org -> jenkins-ci.org
  • @hudsonci -> @jenkinsci
  • http://github.com/hudson -> http://github.com/jenkinsci
  • hudson-dev -> jenkins-dev
  • hudson-users -> jenkins-users
  • hudson-commits -> jenkins-commits
  • hudson-issues -> jenkins-issues

然后把代码fork出一份来(这里好笑的是Hudson与Jenkins都声称对方是自己这里的子分叉,都跟孩子斗气似的),即便分出来了,但是绝大部分还是基于之前的核心代码,所以你可以通过下图看到Hudson与Jenkins的界面都十分类似。

Jenkins的界面

Hudson的界面

但是有一个值得注意的地方就是两个系统的logo,其中Hudson是一个高傲的老头子,而Jenkins是一个谦卑为你服务的老头子。

分家之后,Hudson有Oracle和Sonatype’s corporate的支持和Hudson的注册商标,而Jenkins拥有的是大多数的核心开发者,社区,和后续更多的commit。比如下图是分家之后两个软件的对比。两个软件的活跃程度十分明显,Jenkins遥遥领先。

CI持续集成的工作原理

上面讲完了主流CI工具的江湖故事后,我们来看下这类工具本身的技术情况。其实这类工具的工作原理大同小异,比如下图,一个典型的用例是

  • 程序员在本地开发完成后把代码提交到VCS (Version Control System)比如SVN, Git, Perforce, RTS等
  • CI工具发现有新的check in 自动启动去抓取最新的代码。当然这里有很多不同的配置,比如除了主动监视VSC外,还可以使用CRON等配置按时启动,比如每隔一个小时启动一次,或者每两次check in 启动一次,等等很多的策略。
  • CI可以配置使用集群的编译机器,去选择最合适的机器(有不同的策略,比如找到最清闲或者离代码文件距离最近的机器等)来编译源代码
  • 根据不同的配置,CI有可能会调用配置好的测试用例,如果测试失败,根据策略(比如少于几个错误就先忽略)要么通知用户,要么继续跑测试用例
  • 根据配置,CI可能会去执行其他操作,比如静态源代码分析,如代码有没有不符合公司安全要求,把连接密码写在代码里面等等,还有比如生成文档,测试报告,等。
  • 如果所有定义好的jobs跑完,去生成最终报告并送给用户
  • 生成一些分析报表,比如最近成功率,最近哪些程序员造成的错误最多等等。
  • 一些高级的CI,比如Jenkinsg还支持自定义扩展,也会去按配置去执行。

jenkins-plugin-diagram-saci

这其中如果任何一步出现了错误,比如某个程序员在提交代码时忘记同时提交一个新写的类,造成失败,首先在CI(比如Jekins,或者Travis)上会显示错误 (比如下图),同时还可以配置CI工具会发出邮件提醒,甚至可以根据提交信息智能的显示出来是哪个程序员搞砸的。

总而言之,这个自动化的过程就像是一个可以配置的流水线,在其上可以添加任意个不同类型的节点,在每个节点可以通过灵活的配置来设置需要完成的工作,还提供了统计及报表,邮件通知等功能,方便团队高效的管理软件的持续集成。

发展及未来

目前的CI也在处于高速发展期,比如最新的Jenkins 2 可以支持使用Groovy编写插件,pipeline等。同时也出现了像是开源的__Travis__之类的持续集成service,即你不用自己去安装调试Jenkins,直接写个YAML文件 (.travis.yaml)放到云上,自动就可以使用其提供的服务了。

另外,持续集成也在跟其他新兴技术相结合使用,比如结合云计算及分布式处理,可以提高CI的运行速度和容错能力,比如下图中的各个服务器可以分别使用cluster(集群)而非一台机器,这样就可以避免所谓的SPOF (Single Point of Failure)单点故障。

如果有什么问题或者想要跟我讨论,请通过如下方式找到我。

联系我:

  • phray.zhang@gmail.com (email/邮件,whatsapp, linkedin)
  • helloworld_2000 (wechat/微信)
  • github
  • [简书 jianshu](http://www.jianshu.com/users/a9e7b971aafc)
  • 微信公众号:vibex
  • webo/微博: cloudsdocker

Reference

2022

Linux Tips

Remember, some things have to end for better things to begin.

Back to Top ↑

2021

How to user fire extinguisher

Summary As you know, staff and your safety is paramount. So what if emergency take place, such as fire in office, how to help yourself and your colleagues by...

Deep dive into Kubernetes Client API

Summary To talk to K8s for getting data, there are few approaches. While K8s’ official Java library is the most widely used one. This blog will look into thi...

Whitelabel Error Page

Summary Whitelabel Error Page is the default error page in Spring Boot web app. It provide a more user-friently error page whenever there are any issues when...

Google maps no photos reviews

Summary I found a weird problem of the app Google Maps of my Oppo Android phone. That’s when you search a place in Google map, say “Central Park”, ideally th...

Debts in a nutshell

A debt security represents a debt owed by the issuer to an investor. Here, the investor acts as a lender to the issuer which may be a government, organisatio...

Back to Top ↑

2020

Debug Stuck IntelliJ

What happened to a debug job hanging in IntelliJ (IDEAS) IDE? You may find when you try to debug a class in Intellij but it stuck there and never proceed, e....

Awesome Kotlin

Difference with Scala Kotlin takes the best of Java and Scala, the response times are similar as working with Java natively, which is a considerable advantag...

JVM热身

此文是作者英文原文的翻译文章,英文原文在:http://todzhang.com/posts/2018-06-10-jvm-warm-up/

Mock in kotlin

Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...

Mock in kotlin

Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...

Curl

Linux Curl command

AOP

The concept of join points as matched by pointcut expressions is central to AOP, and Spring uses the AspectJ pointcut expression language by default.

Micrometer notes

As a general rule it should be possible to use the name as a pivot. Dimensions allow a particular named metric to be sliced to drill down and reason about th...

Awesome SSL certificates and HTTPS

What’s TLS TLS (Transport Layer Security) and its predecessor, SSL (Secure Sockets Layer), are security protocols designed to secure the communication betwee...

JVM warm up by Escape Analysis

Why JVM need warm up I don’t know how and why you get to this blog. But I know the key words in your mind are “warm” for JVM. As the name “warm up” suggested...

Java Concurrent

This blog is about noteworthy pivot points about Java Concurrent Framework Back to Java old days there were wait()/notify() which is error prone, while fr...

Back to Top ↑

2019

Conversations with God

Feelings is the language of the soul. If you want to know what’s true for you about something, look to how your’re feeling about.

Kafka In Spring

Enable Kafka listener annotated endpoints that are created under the covers by a AbstractListenerContainerFactory. To be used on Configuration classes as fol...

Mifid

FX Spot is not covered by the regulation, as it is not considered to be a financial instrument by ESMA, the European Union (EU) regulator. As FX is considere...

Foreign Exchange

currency pairs Direct ccy: means USD is part of currency pair Cross ccy: means ccy wihtout USD, so except NDF, the deal will be split to legs, both with...

Back to Top ↑

2018

Guice

A new type of Juice Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice’s @Inject as the new new. You wi...

YAML

Key points All YAML files (regardless of their association with Ansible or not) can optionally begin with — and end with …. This is part of the YAML format a...

Sudo in a Nutshell

Sudo in a Nutshell Sudo (su “do”) allows a system administrator to give certain users (or groups of users) the ability to run some (or all) commands as root...

Zoo-keeper

ZK Motto the motto “ZooKeeper: Because Coordinating Distributed Systems is a Zoo.”

Cucumber

Acceptance testing vs unit test It’s sometimes said that unit tests ensure you build the thing right, whereas acceptance tests ensure you build the right thi...

akka framework of scala

philosophy The actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-o...

Apache Camel

Camel’s message model In Camel, there are two abstractions for modeling messages, both of which we’ll cover in this section. org.apache.camel.Message—The ...

JXM

Exporting your beans to JMX The core class in Spring’s JMX framework is the MBeanExporter. This class is responsible for taking your Spring beans and registe...

Solace MQ

Solace PubSub+ It is a message broker that lets you establish event-driven interactions between applications and microservices across hybrid cloud environmen...

Apigee

App deployment, configuration management and orchestration - all from one system. Ansible is powerful IT automation that you can learn quickly.

Ansible

Ansible: What Is It Good For? Ansible is often described as a configuration management tool, and is typically mentioned in the same breath as Chef, Puppet, a...

flexbox

How Flexbox works — explained with big, colorful, animated gifs

KDB

KDB However kdb+ evaluates expressions right-to-left. There are no precedence rules. The reason commonly given for this behaviour is that it is a much simple...

Agile and SCRUM

Key concept In Scrum, a team is cross functional, meaning everyone is needed to take a feature from idea to implementation.

Strategy-Of-Openshift-Releases

Release & Testing Strategy There are various methods for safely releasing changes to Production. Each team must select what is appropriate for their own ...

NodeJs Notes

commands to read files var lineReader = require(‘readline’).createInterface({ input: require(‘fs’).createReadStream(‘C:\dev\node\input\git_reset_files.tx...

CORS :Cross-Origin Resource Sharing

Cross-Origin Request Sharing - CORS (A.K.A. Cross-Domain AJAX request) is an issue that most web developers might encounter, according to Same-Origin-Policy,...

ngrx

Why @Effects? In a simple ngrx/store project without ngrx/effects there is really no good place to put your async calls. Suppose a user clicks on a button or...

iOS programming

View A view is also a responder (UIView is a subclass of UIResponder). This means that a view is subject to user interactions, such as taps and swipes. Thus,...

Back to Top ↑

2017

cloud computering

openshift vs openstack The shoft and direct answer is `OpenShift Origin can run on top of OpenStack. They are complementary projects that work well together....

cloud computering

Concepts Cloud computing is the on-demand demand delivery of compute database storage applications and other IT resources through a cloud services platform v...

Redux

whats @Effects You can almost think of your Effects as special kinds of reducer functions that are meant to be a place for you to put your async calls in suc...

reactive programing

The second advantage to a lazy subscription is that the observable doesn’t hold onto data by default. In the previous example, each event generated by the in...

Container

The Docker project was responsible for popularizing container development in Linux systems. The original project defined a command and service (both named do...

promise vs observiable

The drawback of using Promises is that they’re unable to handle data sources that produce more than one value, like mouse movements or sequences of bytes in ...

JDK source

interface RandomAccess Marker interface used by List implementations to indicate that they support fast (generally constant time) random access. The primary ...

SSH SFTP

Secure FTP SFTP over FTP is the equivalant of HTTPS over HTTP, the security version

AWS Tips

After establishing a SSH session, you can install a default web server by executing sudo yum install httpd -y. To start the web server, type sudo service htt...

Oracle

ORA-12899: Value Too Large for Column

Kindle notes

#《亿级流量网站架构核心技术》目录一览 TCP四层负载均衡 使用Hystrix实现隔离 基于Servlet3实现请求隔离 限流算法 令牌桶算法 漏桶算法 分布式限流 redis+lua实现 Nginx+Lua实现 使用sharding-jdbc分库分表 Disruptor+Redis...

Java Security Notes

Java Security well-behaved: programs should be prevent from consuming too much system resources

R Language

s<-read.csv("C:/Users/xxx/dev/R/IRS/SHH_SCHISHG.csv") # aggregate s2<-table(s$Original.CP) s3<-as.data.frame(s2) # extract by Frequency ordered s3...

SSH and Cryptography

SFTP versus FTPS SS: Secure Shell An increasing number of our customers are looking to move away from standard FTP for transferring data, so we are ofte...

Eclipse notes

How do I remove a plug-in? Run Help > About Eclipse > Installation Details, select the software you no longer want and click Uninstall. (On Macintosh i...

Maven-Notes

Maven philosophy “It is important to note that in the pom.xml file you specify the what and not the how. The pom.xml file can also serve as a documentatio...

Java New IO

Notes JDK 1.0 introduced rudimentary I/O facilities for accessing the file system (to create a directory, remove a file, or perform another task), accessi...

IT-Architect

SOA SOA is a set of design principles for building a suite of interoperable, flexible and reusable services based architecture. top-down and bottom-up a...

Algorithm

This page is about key points about Algorithm

Java-Tricky-Tech-Questions.md

What is the difference between Serializable and Externalizable in Java? In earlier version of Java, reflection was very slow, and so serializaing large ob...

Compare-In-Java

Concepts If you implement Comparable interface and override compareTo() method it must be consistent with equals() method i.e. for equal object by equals(...

Java Collections Misc

Difference between equals and deepEquals of Arrays in Java Arrays.equals() method does not compare recursively if an array contains another array on oth...

HashMap in JDK

Hashmap in JDK Some note worth points about hashmap Lookup process Step# 1: Quickly determine the bucket number in which this element may resid...

Java 8 Tips

This blog is listing key new features introduced in Java 8

Back to Top ↑

2016

Java GC notes

verbose:gc verbose:gc prints right after each gc collection and prints details about each generation memory details. Here is blog on how to read verbose gc

Hash Code Misc

contract of hashCode : Whenever it is invoked on the same object more than once during an execution of a Java application, the hashCode method must consis...

Angulary Misc

Dependency Injection Angular doesn’t automatically know how you want to create instances of your services or the injector to create your service. You must co...

Java new features

JDK Versions JDK 1.5 in 2005 JDK 1.6 in 2006 JDK 1.7 in 2011 JDK 1.8 in 2014 Sun之前风光无限,但是在2010年1月27号被Oracle收购。 在被Oracle收购后对外承诺要回到每2年一个realse的节奏。但是20...

Simpler chronicle of CI(Continuous Integration) “乱弹系列”之持续集成工具

引言 有句话说有人的地方就有江湖,同样,有江湖的地方就有恩怨。在软件行业历史长河(虽然相对于其他行业来说,软件行业的历史实在太短了,但是确是充满了智慧的碰撞也是十分的精彩)中有一些恩怨情愁,分分合合的小故事,比如类似的有,从一套代码发展出来后面由于合同到期就分道扬镳,然后各自发展成独门产品的Sybase DB和微...

浅谈软件单元测试中的“断言” (assert),从石器时代进步到黄金时代。

大家都知道,在软件测试特别是在单元测试时,必用的一个功能就是“断言”(Assert),可能有些人觉得不就一个Assert语句,没啥花头,也有很多人用起来也是懵懵懂懂,认为只要是Assert开头的方法,拿过来就用。一个偶然的机会跟人聊到此功能,觉得还是有必要在此整理一下如何使用以及对“断言”的理解。希望可以帮助大家...

Kubernetes 与 Docker Swarm的对比

Kubernetes 和Docker Swarm 可能是使用最广泛的工具,用于在集群环境中部署容器。但是这两个工具还是有很大的差别。

http methods

RFC origion http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.1.2)

Spark-vs-Storm

The stark difference among Spark and Storm. Although both are claimed to process the streaming data in real time. But Spark processes it as micro-batches; wh...

微服务

可以想像一下,之前的传统应用系统,像是一个大办公室里面,有各个部门,销售部,采购部,财务部。办一件事情效率比较高。但是也有一些弊端,首先,各部门都在一个房间里。

kibana, view layer of elasticsearch

What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...

kibana, view layer of elasticsearch

What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...

iConnect

UI HTML5, AngularJS, BootStrap, REST API, JSON Backend Hadoop core (HDFS), Hive, HBase, MapReduce, Oozie, Pig, Solr

Data Structure

Binary Tree A binary tree is a tree in which no node can have more than two children. A property of a binary tree that is sometimes important is that th...

Something about authentication

It’s annoying to keep on repeating typing same login and password when you access multiple systems within office or for systems in external Internet. There a...

SQL

Differences between not in, not exists , and left join with null

Github page commands notes

404 error for customized domain (such as godday) 404 There is not a GitHub Pages site here. Go to github master branch for gitpages site, manually add CN...

RenMinBi International

RQFII RQFII stands for Renminbi Qualified Foreign Institutional Investor. RQFII was introduced in 2011 to allow qualified foreign institutional investors to ...

Load Balancing

Concepts LVS means Linux Virtual Server, which is one Linux built-in component.

Python

(‘—–Unexpected error:’, <type ‘exceptions.TypeError’>) datetime.datetime.now()

Microservices vs. SOA

Microservice Services are organized around capabilities, e.g., user interface front-end, recommendation, logistics, billing, etc. Services are small in ...

Java Class Loader

Codecache The maximum size of the code cache is set via the -XX:ReservedCodeCacheSize=N flag (where N is the default just mentioned for the particular com...

Back to Top ↑