Continuous deployment and flexibility

16 03 2010

Say a client asks you to build an application such that he can change it between releases. He wants to be able to do this so that he can react to business needs quickly. But most of the time this is interpreted as a way to build an application that is highly configurable.

One of the application I was working on had very high levels of complexity because of the infrastructure to make it configurable. The client was using a very expensive rule engine which was sold to him saying that business can deploy rules without developer help.

Clearly what the client wanted here was to be able to react to business needs. But what he got was an application which had a baggage of infrastructure to make it highly configurable. In this day and age where many projects use continuous deployment, there is not much of an issue in responding to a business need. Release becomes a non-event.

Even during maintenance phase, we can continue to keep the continuous deployment setup in some cases.

It is just a thought and I would love to see what people have to say about this.





Building java projects on runcoderun.com

26 01 2010

RunCodeRun is a hosted Continuous integration environment developed by Relevance. Thanks to the Relevance team and their effort to provide this service free for opensource projects.

I wanted to build rapa on RunCodeRun. I was trying to setup my project on RunCodeRun using instructions in a blog post. It is a very useful blog and easy to follow. But I was having trouble getting code to compile on runcoderun box. Here are the problems and their solutions.

I guess there is no real jdk on runcoderun, so my compile task on ant would not work.

I checked in tools.jar as a part of my project libraries. Then I added it as part of the classpath when invoking ant from rake.

classpath = [File.join(".","lib","ant.jar"), File.join(".","lib","tools.jar"), File.join(".","lib","junit.jar"), File.join(".","lib","ant-junit.jar"), File.join(".","lib","ant-launcher.jar")].join(File::PATH_SEPARATOR)
system "java -cp #{classpath} org.apache.tools.ant.Main -emacs dist"

Even after I fixed that the build would always go green even if there was failure from ant

I guess the exit status returned by rake was always zero irrespective of what ant returned. I changed the rakefile as shown below.

system "java -cp #{classpath} org.apache.tools.ant.Main -emacs dist"
exit $?.exitstatus

This seemed to fix all the issues with the build.

Along with these issues I was also facing an issue with ant-junit task with junit 4. Basically standard distribution of ant does not understand the annotations of junit4 and requires the test class to extend TestCase. I had to checkout the latest source from ant subversion trunk and build it. I am currently using these jars.

Have a look at the project in github.

http://github.com/harikrishnan83/rapa

Continuous integration on.

http://runcoderun.com/harikrishnan83/rapa





Mnesia Quickstart

5 01 2010

Basic introduction on Mnesia

Mnesia is a RDBMS. But it belongs to the nosql category of RDBMS. Reason being the query language is not sql but Erlang. That makes it very easy to store and retrieve without having to go through Object Relational Mapping. So we can actually call Mnesia an object relational database.

Why and where would one want to Mnesia?

Erlang in general is used to program highly distributed and fault tolerant systems. Even though it has its roots in the telecommunication industry, it has proven useful in several other sites like ejabbered, Facebook chat etc. Mnesia is just a part of Erlang and is built with Erlang.

Hence it gives you configurable degree of Fault-tolerance (by means of replication).

Another important feature of mnesia is the Dirty read interface. It is possible to read, write and search Mnesia tables without protecting the operation inside a transaction.

Quickstart

The below steps should get you started on mnesia, if you are using it for the first time.

Required Software – Erlang

I use ubuntu as my OS. But that should not make it much different on any other OS.

Now lets start with some code. It is useful to install the table-viewer (erlang-tv).

The goal of this exercise is to create table called person, insert few records and read them.

create a file called Person.hrl

-record(person, {name,      %% atomic, unique key
age,        %% age
married_to, %% name of partner or undefined
children }).%% list of children

create a file called Person.erl

-module(person).
-include("<path_to_person.hrl>/person.hrl").
-export([init/0]).
-export([insert/0]).
-export([read/1]).
init() ->
mnesia:create_table(person,[{attributes,record_info(fields,person)}])
.
insert() ->
T = fun() ->
X = #person{name=john,
age=36,
married_to=ana,
children=[josh,kelly,samantha]
},
mnesia:write(X)
end,
mnesia:transaction(T)
.
read(Name) ->
R = fun() ->
mnesia:read(person,Name,write)
end,
mnesia:transaction(R)

Start command line erlang. Type in the below command from the directory which contains the above two files

erl mnesia dir .

The above command conveys that the current directory will be used to store Mnesia files.

Compile the person.erl

>c(person).

{ok,person}

Start mnesia

>mnesia:start().

Create person table.

>person:init().

Insert a record.

>person:insert().

Use table viewer to check if the record has been inserted.

>tv:start().

This will launch table view application. By default the table viewer shows the ETS tables. To look at the table we just created go to view menu and select Mnesia tables.

Read the record using Mnesia:read()

>person:read(klacke).
{atomic,[{person,john,36,ana,[josh,kelly,samantha]}]}

In my next post I will cover Mnesia queries as List Comprehension.





Remote inception – An Experience Report on an inception over phone

9 12 2009

Before I start, I would like to state that this article does not advocate for or against running an agile inception over phone. It is more of an experience report. Please feel free to post your comments.

Introduction

Inception is at the heart of a successful agile engagement. In an agile project we work with the client and not just for the client. Inception starts the process where the team, client and consultants, start thinking alike and working together.
It is so much easier to work with a colleague once you have synchronized your frequencies/wavelengths. Okay, enough blabber about wave theory.

This article is about my experience, learning and rant about remote inceptions. I intend to keep it more like a free flowing conversation.

Inception

Projects are set fail if the initial understanding and the basis for further development is flawed. Some key questions that immediately come to our mind:

Does the client know what he wants?

Does he really need to build it, or can he buy something that already exist (COTS)?

The answers to these and many other questions would become apparent in an inception.

Ideally inception is where we start by setting a vision for the project, break it down into achievable milestones and further down into playable stories. But all this mandates that you have the client right in front of you.

Most inception exercises require face to face interaction to make communication as clear as possible. It is necessary to use tools (simple and sophisticated) to make mental model explicit, elicit the requirements and clear any doubts. Inception is a fun and effective way to interact with the client and bring every one on the team on board with the project’s goals.

Most inception should have the below activities in the agenda.

1. Team introductions – May seem simple. But simple activities like playing a small team game act as the all important ice breaker.

2. Collaborative Modeling sessions – As many or as little, as per requirement of the project. A good inception would have several of these sessions on project specific topics as well as general discussions on Non Functional Requirements.

3. Prioritization – Lay out the options in front of the client story cards. Let the client move the cards to prioritize them. In some cases this exercise leads to a rough release plan.

4. Inception showcase
The above activities are a small subset of an inception. But, these are the ones which bring out the most useful facts that are necessary for the project success. Also they are the ones which require as much face to face interactions and team efforts.

At the end of an inception the team must be able to decide if they should go ahead with the project.

Context

Let me now explain a little bit about the scenario we were faced with. Our clients had very limited budget and could not afford to include travel expenses for either them to travel to our location (India) or for us to travel to theirs (Chicago). It may seem very sensible to not start the project until sufficient budget is available. But the client could not get more budget unless something was built and built soon. So we had to do an inception with them over phone, with a 12 hour time difference (Sadly the video conference equipment on their side was broken).

All this got us be more resourceful and improvise with what we had. The only way ahead was to address all the risks as best as we could.

Managing risk

A remote inception is very risky business. The probability of success is quite small. Always communicate this to the customer and try and push for a face to face inception. Remember this is not for your benefit, but it is in the best interest of the customer. It is a good idea to maintain a shared risks log with the customer.

Below are some risks that we faced.

Risk: Understanding about some Features may not be completely correct

Mitigation:

  • Client was made aware that there may be minor misunderstanding despite best efforts.
  • In our case the application functionality was quite closely associated with the UI. So we came with early mockups that were as close as possible to what the client wanted. We let them edit the same and maintained them for future reference.
  • Rather than plainly documenting technical understanding, we built very crude prototypes. Most of the time, code is the best documentation and communication mechanism.
  • The client was made aware that our initial estimates would be bumped up by a certain risk factor to accommodate any issues with understanding. It is better to promise less and deliver more.

Risk: 12 hour time lag. It was imminent from the beginning that we had to spend time outside our usual working hours to spend enough time on the inception.

Mitigation:

We scheduled for calls which ranged between 3 to 4 hours everyday. A face to face inception can have day long agenda. But it is better to maintain lesser number of hours on remote inceptions. Small 15 minute breaks were counted in.

Inception Agenda

We made sure that all stake holders were in a position to dedicate time for inception. Instant messenger proved very useful. We also sent out links to tools like webex (Desktop sharing tool). Initially we were confident about our own superhuman capabilities to spend late hours at office to have longer conference calls. But a senior member in our team rightly pointed out the flaw and reduced it to an optimal 3 hour call. This suited us well. After three hours over the phone it is extremely tiring to do any other productive work.

We prepared the agenda to ensure that we had time to cover all topics that we considered necessary. But it was not something that was set in stone. Some sessions finish ahead of time while others may reveal unknown areas, which require fresh slots to be included. We revised the agenda from time to time.

A typical day would start with the recap of the previous days meeting notes. This would be quick 15 minute exercise which would warm up the team for the long call ahead. We also used this time to follow up on each other’s progress.

Communication and meeting notes

As in the case of any normal inception, never go without a good scribe. We took turns at this role and noted down all the key points. Though it may have sounded silly sometimes, we tried to paraphrase the client’s sentences and validated our understanding. The client was informed at the very beginning of the inception that we may have to repeat some lines to confirm our understanding. In our case one team member from our customer side volunteered to take notes as well. At the end of the day we would share notes and if there are any differences in understanding we would resolve it in the following day’s meeting. Once all differences are cleared, we would put it in a place where every one has access.

Try to learn how each person sounds, so that you can associate a voice and/or accent to a person. Also suggest your customer to do the same. This helps a lot in keeping the conversation easy.

Sometimes you will not know when the person on the other side of the phone has lost interest in what you are saying. It is better to speak slowly and clearly. While talking to a person face to face it is very easy to detect when he/she is loosing interest. On the phone the one possible way to do this is to include small questions while one speaks. This way you know the person on the other side is listening.

Tools

We used low tech tools to simulate a virtual card wall where clients to could move cards. You could use an online card wall for this. Screen sharing tools like webex are extremely important. There are quite a lot of free tools available.

Start using a project management tool early in the cycle. Start adding stories to the project management tool as early as possible. A spreadsheet may be easy to start with. We used mingle for project management.

In Retrospect

If I had to do this all over again, I will still consider it extremely risky business. Few things that I might do differently are listed below.

  1. Get the video conference equipment worked out early. In our case since the team size was small. So this did not become a great issue. I would strongly recommend having a video conference for bigger teams.
  2. If there is not enough budget for the entire team to travel, try to have at least one representative from your side at the customer’s location. He could facilitate the activities.
  3. Capture the clients mood over the period of inception using tools like Niko-niko Calendar

Summary

Remote inceptions are tough if not impossible. Try to avoid it as much as you can. But if have to do it, you know are not alone. In the end it is our goal to help the customer, no matter what the constraint. Fortunately, in our case the exercise was a success and the customer was happy.





Testability Explorer and rapa

5 09 2009

After a hard day at work my mind was in no state to churn out quality code on open source. But really wanted to get something done about rapa. So thought it might be a good idea to integrate some metrics into the build and see what they had to say.

Misko has created this really cool tool called Testability Explorer. It tells you how testable your classes are. But why do we need such a tool?

Rapa was mainly designed and developed the TDD way. There is always more to learn and improve in the code. Certain parts of Rapa became increasingly difficult to develop the TDD way. We also need a tool that can Flag some basic testability issues.

Background

Lets start with some basic testability issues that we all may have faced at some point in time.

A very common issue most people new to TDD and mocks face is the mocks returning mocks. Below is an example.

when(mockMethodProvider.getMethod()).thenReturn(mockMethod);
MethodExecutor executor = new MethodExecutor(mockMethodProvider);
executor.execute();
verify(mockMethod).execute();

In the first line we are setting an expectation on mockMethodProvider to return a mockMethod. The mockMethodProvider is injected into the methodExecutor. Then we call execute on executor. In the last line we are verifying that execute was called on the mockMethod by the executor.

If we look at this code closely we can observe that MethodExecutor is being injected with mockMethodProvider when all it needs is the mockMethod.

It is a simple design issue. You can find this and many more examples of such common issues with testability in Misko’s blog.

Rapa and Testability Explorer

Integrating testability explorer itself into the build is an easy process. I used the instructions in this link.

Below is the report.

Testability Explorer Report for Rapa release 0.8
Testability Explorer Report for Rapa release 0.8

Looks like the code is not too bad. 🙂





altnet 2009

2 08 2009

I reached there on time for a change. Today was the third day of the event. Big thanks to ian, alan, ben and seb for organizing the event. The event started out with the organizers explaining the the open space format. The conference started with people signing up for various topics and putting it up on the chart. Twitter channel was continuously abuzz with information.

A talk about Concept to Cash / Building software that matters, brought up issues like making sure there is business value in what we build. Gojko did a very good job facilitating this discussion.

There was a discussion about Domain Events based on Udi Dahan‘s blog. Turned out to be a real crowd puller.

Also managed to be a part of informative discussion about REST.

A group of people gathered together for lunchtime discussion on real work issues adopting agile. They identified issues like working with distributed teams.

    I put up a topic on “Concurrency paradigms”. The conversations revolved around the three most popular Concurrency paradigms locks, actor model, and shared transaction memory. It was great learning experience talking to the guys at altnet. We discussed about real world usages of these paradigms. Consensus was that if we are solving a simple concurrency problem then we can still use threads with locks. If we need high levels of concurrency then we have to do away with locks. Actors based concurrency is a good idea for this. But if we need concurrency and also need to take care of transactions then we can use shared transaction memory. More on this in this blog.

    The other topics that were discussed are available on the altnet site

    I have uploaded some pictures in picasa.

    Apologies for an unorganized post. I will edit this post when time permits.

    Alt.net is by far one of the most enthusiastic event I have ever attended. Great job guys to all who were part of this. Thanks to all the wonderful people I met at this conference and the great memories I took away.





    Document Oriented Databases

    10 05 2009

    History

    Relational Databases have been the almost the only way applications persist data. In the days when code was written mostly with COBOL even navigational databases were sufficient. The switch to relational databases made it easier to query. Not much has changed in the way we store data since then. This may be attributed to fact that query performance is still the most important thing.

    Object oriented code as we all know has to go through mapping tools to be persisted as relational entities. Are we going to use the same database concepts in the coming years?

    Retrospect

    Have you been bothered by the below issues?

    1. We model business logic as interaction between objects. Concepts such as triggers also model some amount of business logic. Though the confusion should not arise, many a times people mix business logic in database layer and business layer. In some rare circumstances there may even be duplication of logic in Business layer and Database layer. Example: When a new customer is created, a trigger inserts a new row into another table called privileged customer based on a data condition. The trigger here has business logic in it which is not covered by unit test cases.
    2. Applications have Domain validations like customer name cannot be null. If you have such validations in your code, which we normally do, then why do we need a database that also does these validations?
    3. We create great Object oriented business layer and lose sleep over mapping them to a relational model. All I care about is persistence of the state of my objects. Though I agree relational databases have very good query performance, are we really keeping our eyes closed to other persistence techniques?

    Alternate Approach

    CouchDB – A Document Oriented Database:

    Document-based databases do not store data in tables with uniform sized fields for each record. Instead, each record is stored as a document that has certain characteristics. Any number of fields of any length can be added to a document.

    CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API.

    All that apart Couch DB is not an object-oriented database.

    In a relational database we would store addresses of a customer in a different table, with a foreign key linking it to the customer.

    With a document-oriented database, such as CouchDB, the nested resources maybe stored together with the main resource. Example JSON Document:

    {
    "name": "Geeky Customer",
    "adresses": [
    {"street": "Wall Street", "Number": "2"},
    {"street": "Dalal Street", "Number": "4"},
    ]
    }

    This brings us to an interesting thought. We do not require ORM frameworks like hibernate, active record which are mostly written around SQL-like problems that CouchDB just doesn’t have.

    There are a libraries like CouchRest, RelaxDB, ActiveCouch etc which provide simple ways to connect to CouchDB.

    I have taken CouchDB only as an example. There are many new database which are quickly becoming popular for specific situations.

    We do not have to choose a database because it has been chosen by the majority. It may be worth the effort to get up from your couch and take a look at other databases.