I am now officially almost at the halfway point of the first phase of a software upgrade on production system.
As it is an SOA application we took a different approach than most people were pushing for and it is interesting to be here supporting the team as they go through the throws of wrestling the cats back to the herd. It is a small team, but they are in Edmonton, Halifax, Montreal, Markham, and Mississauga and they are doing a good job of managing through this.
The secret seems to be as you would guess in the people and the approach we took. For each of the main tasks we have a buddy system going leveraging some good stuff from Agile.
The database server upgrade team had three main players with one on call and two working together to follow the process through. They patiently walked through a process they rehearsed in development twice, and they worked perfectly together. They were able to catch the first issue ahead of any impact and then they captured two more afterwards. Meanwhile working through the unknown of what would happen based on production data. It appears that the horsepower of the production server seemed to make up for the increase and difference in the development data. This was the evolution of desk checking at it's best as one of the guys had actually walked through the others code ahead of time (last couple of days) and then was able to tell when we would be past the biggest bumps.
The application server was the surprise, we are still using the same process, 3 people, two on-site with another on call. When we started we opened a bridge for voice and the main control developer shared his desktop over the internet. What I ended up seeing is that often someone observing was often looking at the screen differently and as long as the communication was good, they jumped ahead of issues.
The biggest thing I see is a group of people who are singularly focused and energized by the goal. Even Mr. Murphy shows up they are able to overcome.
Man this is fun.