Almost all software that has every been built had to go through an integration stage in one way or another. It is quite odd that many software projects even to this day look at integration as a separate phase in the project. But things are changing quite fast and many projects adopting Continuous Integration. This is good news because we are saving ourselves a lot of time wasted in the so called integration bugs.
So what is the issue?
Lets cut to the chase. Mostly organizations tend to try out CI on a not so important team before it is adopted across all teams. But in such a scenario CI is only building a smaller system. Even organizations that have been using CI for quite some time seem to have CI builds per team. But usually the software that the smaller teams are building are just small pieces in a bigger system. Integration bugs still have a longer feedback cycle.
It is not enough to have a CI infrastructure that only verifies the subsystem. CI must deploy the smaller module on to an environment with the rest of the system and run tests as a whole. This gives feedback on the integration.
So we get the point. Whats the big deal?
Actually setting up a good CI is not so simple. Unless the build system is not well thought after CI does not come for free. While writing the build scripts/code one must keep in mind the bigger picture. The effort involved is very similar to a traditional production release. It is also important to revise the build from time to time. While a build that fails for several unknown reasons is a pain, a build that is not proving anything is even worse.
CI should also not be an activity solely done by configuration management team. The people who are responsible for the application design also have a big role to play. One size does not fit all and concepts that proved successful in one project may not work so well for another. Also CI has to evolve with the system that it is integrating. Unless it is up to date with the latest design changes its not doing much.
If your CI is testing a subsystem it can only be called an automated build system for that module. CI needs to verify key integration issues and to some extent even performance. Writing build systems that truly integrate is an interesting activity. It gets you thinking on how two systems integrate. Questions that you would have not thought about previously suddenly become more obvious.
A Continuous Integration build must be run on a production like environment. If not, the application must be deployed to production as often as possible. A well written CI also inspires confidence to move to a continuous deployment mode.
I do not claim to be an expert on CI, but I think it is important not to forget the principles behind practices. I would associate more importance on the integration bit in CI than smaller checks that verify code style etc.
Also I would recommend reading Sai’s blog on CI.
Let me know your thoughts on this.
HariKrishnan 
Nice and clean post. But I have to disagree with certain things that you say.
Actually writing a good CI is not so simple.
That’s where what you say shows your disconnect. It’s not about writing, it’s about implementing.
And how much production like an environment should I have? It is not always possible to mirror the production environment exactly. Everything from the build times to the feedback loop to the hardware constraints come into play.
It is great if the CI is all covering. It is pragmatic to make sure that the CI tells you what you think it tells you so that you can be prepared for the eventualities in your application. You know what your CI did not cover. What it’s limitations were and you are prepared for it.
It is a concern when your CI lies. Integrity is more important than integration.
Good point. If you know what you are not covering as part of your CI, you will be prepared to face it. Also you may try to cover these areas eventually. But to trust a build system that only provides subsystem level coverage for module health and ignore the integration troubles may be disastrous. Thanks for the comment. Appreciate it.
I recently touched on this in my talk at http://www.devopsdownunder.org this month.
It really depends on what sort of business model your selling your software through – are you building SaaS? Then yeah – you should be isolating out each build into a copy of your production environment and integrating there and honestly there is no good excuse for not doing so because modern tooling is already in most cases here.
Systems configuration management tools like Puppet make reusing your production environments during integration really easy – the same Puppet production configuration can be mixed in with test specific configuration (for example, use of fresh new databases instead of your live production data) to produce an environment where you can do this every build by utilising virtualisation.
Unfortunately some platforms (OK, windows) make this a little more difficult because the tooling simply does not exist for configurations that have a lot of complex dependencies and you often have to resort to building and maintaining pre-prepared virtual machines for the purpose.
However if your in the business of selling ‘Behind the firewall’ (BTF) software the situation can be vastly different – you never really know what your customers are actually running your software on and in what configuration. BTF is costly – in SaaS your only testing a single environment – yours – and that drives the cost of testing your software way, way down.
With BTH you really need to pick your battles during development and be asking yourself “What sort of customers are costing us time to support? Oh, we seem to be getting a lot of support issues for customers with Windows 2003 and SQL Server 2000?” and change your testing and integration strategies accordingly.
A lot of problems I’ve seen with smaller teams who integrate into a larger product (We have a product called JIRA Studio which is a SaaS bundle of our BTF Wiki, Issue Management, CI and code review tools that run into this exact problem) just don’t simply have the confidence to say _when_ they able to integrate with other teams.
I think the lack of this confidence is a huge part of where modern CI systems fall down and simply become glorified test reporters in the eyes of development teams.
Looking forward to hearing your thoughts.
Very interesting points. IMHO the exercise of building a CI system that actually integrates subsystems and tests them as they will run in production, irons out many issues. It even clears our thought on the design of the application at times. As part of this we are also making the effort of automating the deployment and in effect getting ourselves prepared for the production release. It is called CI for a reason. :).
Thanks for your comment.
I don’t see a disconnect here. Writing good CI is very challenging and implementing good CI is very challenging. If one is working with just one technology – say a pure Java or a pure C# house – it is fairly straight forward; CI in a mutli-technology system, especially cross platform, requires a huge set of build and test scripts (or what ever test system you have) to be implemented and then maintained as the system under CI develops.
My recent experience is that CI is getting more costly (time/resource) as software is getting more complex. However, its pay-off is enormous. CI _is_ the way to go but we have to factor in the costs up front.
One thing that I have noticed is that quite often unit testing and CI get mixed up in people’s heads. Then CI becomes ‘continuous unit test’ which is not useless but does represent only a very small part of the challenge space.
In my opinion if we are building a CI system, it better be doing what it claims. If not at least the team should be clear on what is not covered. It is dangerous when people mix up unit testing and CI. I think you put it in a much better way than I did. Thanks for bringing this up.
As for the cost associated with building CI, I think in the long run it pays for itself.