Building a successful front-line Ops team

Okay, this has turned into a mini-novel, but I think these are all important points. There are a ton of others to consider, but I don’t want this to turn into War and Peace. 🙂 While the examples are ops-focused, I suppose most, if not all, of these points can apply to team building in general.

#1: Understand your team’s work load and type
Before you build a team, you really need to know what you’re trying to address by hiring a team. How can you build a balanced and ‘right-sized’ team if you don’t know what your work load looks like? Say you’re hired and your mandate is to hire four senior systems engineers to fill out a new front-line operations team. You have just 500 machines, less than 20 incoming requests per day, 60% of your work load is deploying software (running a series of scripts) and another 25% is filled mostly with rebooting machines. Doesn’t hiring 4 senior engineers seem like a bit much?

#2: Hire for the level of work you currently have
It can be really tempting to over-hire for the role(s) you’re trying to fill- especially if you’re given the green light to “go hire rock stars”. But what happens if you hire a rock star but don’t have the senior-level work to keep them challenged? So in the scenario above, what if you hire a couple of junior engineers who can cover the ‘grunt work’ but who have a lot of potential, and supplement that with one senior engineer who can code improvements in the deployment system and create some tooling around host reboots? Face it: if you hire a senior systems engineer to reboot machines all day with no time allowed to automate that work away, he/she won’t stick around for long. A typical onboarding process can take four weeks or longer to complete, and it usually takes a new hire three months (-ish) to feel moderately comfortable in their role. That’s a lot of time to invest in someone just to do it all over again, not to mention the disruption in team cohesion/morale and the group’s road map.

#3: Hire engineers who can help themselves but who still have runway to grow
This goes hand-in-hand with my points above. Don’t forget that if you’re hiring a front-line team, you’re most likely hiring more junior engineers who need to have a development plan. They need work that’s challenging, but not *too* challenging. It’s a delicate balance, but hiring someone who’s a novice at system administration to reboot machines and only reboot machines for their entire career isn’t doing anyone any favours. You’re hiring ‘go-getters’ though, so you may have to reign them in or point them in the right direction.

#4: Hire for heterogeneity
I’m a strong proponent of not building a team of people who all have the same skill set for a few reasons. First, you could wind up with gaping holes in knowledge (ie all systems engineers who have no networking experience). Secondly, each person by virtue of having a different skill set will have built-in opportunities to mentor the other people on the team. Thirdly, people who don’t have the skill set will have an opportunity to learn something new. Fourth, it’s good to have differing perspectives on issues so you can explore all options for solutions to issues. Fifth, you need to have room for people to grow and get promoted at varying rates. Lastly, it’s my personal opinion that a modicum of disparity in skill sets is healthy- it keeps people on their toes and promotes healthy competition if managed properly. (reward not only the technical achievements but the ‘softer’ skills like process improvements and customer service, and you should wind up with healthy peer pressure)

#5: Have a detailed training plan for your n00bs
It’s not just about technical training, although understanding the architecture and the technologies the company employs are both very important. New team members need to understand tools, processes, and the major players/teams involved in each of the large buckets of work/requests they’ll come across. Putting some thought into the plan up front will actually decrease the time it takes to get someone up to speed. Start with the fundamentals; explain typical requests and accompany that discussion with hands-on training for each task. Pair each new hire with a mentor who’s completed the task before. Knowledge transfers on specific architecture components by subject matter experts may also be helpful, but make sure that you time those sessions so that new hires aren’t too inundated with information that they can’t immediately apply.

#6: Measure your work load/effectiveness/progress
If you’ve taken care of point #1, then you should be able to measure your effectiveness. For an Ops team, this shouldn’t just cover basic ticket metrics like number resolved and time to resolve. It should include the type of issues coming into your queue (success is seeing a downward trend in tickets relative to your drivers, which could be number of machines you support, number of changes in the environment, etc), improvement in basic measurements like site/service latency or errors/timeouts, and customer satisfaction. That last metric is pretty difficult to measure without canvassing customers directly, but proxy metrics like number of tickets reopened for the same issue or total contacts per issue are a start. My favorite metrics are the ‘types of issues coming into the queue’, and I’ve had some success in prioritizing our team’s work load to drive down those interrupts that hog the greatest amount of time or effort. Having short- and mid-term goals for efficiency gains and reviewing progress against them in team meetings shows 1) that team members’ work is making a difference and 2) that the ‘crap’ work is going the way of the Do-Do, freeing up the team to work on increasingly challenging issues, which means that they get more growth opportunities. It’s a virtuous cycle. It does take some time to train up new hires, and if you’re starting from scratch with a new team, you may not see a great strides in metrics to begin with, so make sure to temper your goals according to where your team is at in its evolution.

#7: Sell the team to your partners and customers
If this is a new team, you’re probably going to have to play the role of promoter, even if everyone in the entire company is on board with your charter. Point #6 will certainly enable this, and as long as you’re making progress toward your goals and showing value to the rest of the organization, this should just feel like part and parcel of your job as the leader of your team.


4 thoughts on “Building a successful front-line Ops team

  1. I’m astonished, I have to tell you. It’s not every day you come across a blog that’s this good on web 2.
    0 site, and you’ve got it down perfectly. This matter is an issue that not enough folks are talking intelligently about. Right now I’m very happy I found this throughout my search for anything relating to this.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s