Introducing a New Project (to Ops)

Everyone owns slightly (or drastically) differing opinions about how an IT project ought to be managed. I don’t believe in only one correct way to run a project. The best project teams I work with or observe adapt in various ways: project by project, team by team, customer by customer. Regardless of the method, I expect everyone involved in a project to share some core principles:

  • Adhere to Defined Roles
  • Adhere to Defined Processes
  • Communicate

I’ll focus on the introduction of a hypothetical project of some size and breadth to illustrate how these principles should be applied to make the most of the project team at the outset. The first few days or weeks are the most critical for setting the tone and laying the foundation for the remainder of the assignment. For purposes of this scenario, the production operations organization as a whole is involved in 20+ disparate projects this year, and every one of them is “high priority”. At best, we have two to three engineers focused on specialized areas, from networking to storage to infrastructure core services (LDAP, DNS, etc.).

The introduction of a project is vitally important to its success, but it can be marginal at best without a formal process. Assuming there is no well-defined process, I can learn about new demands on my team in myriad ways, including

  • hallway conversations, either directly or overheard
  • random mention in an email
  • second-hand through a member of any number of non-Ops teams
  • request for resources to help with a build-out within a ticket to an engineering queue
  • random entry in a project plan
  • buried in a response to a separate, unrelated query

None of these can or should be considered a “process”.

Introduction of the Project

Here is how this particular project makes its way through the Ops organization. At this point in the project, a commitment for delivery in two months’ time (40 working days) has already been made to the customer, and the PM has already furnished an update to the customer and senior management that the project is on track. No project plan or list of requirements has been furnished to our org.

During this same time frame, our team is on the hook for delivering two other high-priority projects. Our resource plans show that we’re already 25 man days in the hole without considering this new request, and various engineers have been asked to put in extra time to get us over the hump. We’ve also learned that the project was first introduced to the software development organization four weeks ago.

  • Initial contact with the team comes from three sources: an URGENT email to me from a PM asking for resources to build out a test infrastructure (this indicates it’s already an escalation), a second request from the same PM directly to an ops ticket queue asking for work to be done on the project, and a hallway conversation between one of the Ops managers and a developer on the project with a heads-up that development work is already underway.
  • Early estimates provided to Ops come from software developers who don’t have the time or background to understand our infrastructure or support services well enough to estimate what needs to be delivered.
  • In the next Ops Management meeting, we assign a senior engineering resource to participate in design conversations with the lead developers, and a resource to build out the test architecture once we’ve vetted the design. (the request to build out a test platform makes no sense when we don’t even understand whether the design itself will work in our infrastructure).
  • The scheduling of meetings between the engineers and the facilitation of the discussions falls on our management team. One of our managers will now slip one or more of his own priority deliverables in order to take on this portion of the project.
  • After our engineers complete their design review discussions, it is apparent that the initial resource estimates based on the information we’ve received from the developers are woefully inadequate. The project requires 3 full man weeks of Network Engineering time for design and implementation, a storage engineer for a full month, four weeks of front-line Ops work for deployment of infrastructure, software and testing, and a week of a capacity planning engineer’s time.
  • The project also requires additional hardware which was not factored into our recent server purchase, so we must determine whether we can re-coup machines from other projects or place an expedited buy with one of our vendors. If we determine that we need to “steal” hardware from another project, we add another two weeks of effort from our front-line Ops team.
  • In the middle of trying to work out whether we have enough resources to hit the already-promised deadline, I receive an escalation from the PM for resources to build out the test infrastructure, and am pinged by my boss on why we don’t already have a solid Ops project plan for the initiative. It’s kind of getting ridiculous at this point.

This project is clearly not on track, and due to lack of proactive communication and solid planning, Ops is the blocker right out of the gate. (and you wonder why Ops engineers tend to be surly!) The exercise above has taken five days to complete, which means we’re now down to 35 man days to meet the deadline committed to on our behalf. There is zero chance that we can deliver on this project without missing deliverables on at least one of our other high-priority projects.

Lessons Learned

Based on the core principles listed in the introduction, we have a number of items of feedback for various parties involved in the project.

Adhere to Defined Roles

  1. The roles and responsibilities for everyone involved in the project weren’t clearly defined, including determining the correct points of contact and decision makers within Ops. This led to the PM contacting both me and an engineer for the same work, as well as a developer pinging another Ops manager.
  2. Because there was no single owner, we risked either duplication of effort or no effort, both of which are inauspicious ways to kick off a project.
  3. Ops’ absence in determining the delivery dates for the project meant that the customer received a promised completion date that the project team could not honour.

Adhere to Defined Processes

  1. Multiple processes were used for introduction of the project into Ops.
  2. Because there was no agreement on how to introduce the project into our organization, I received an escalation contact prior to an initial request for resources.
  3. No regular update was given over the first four weeks of the project; Ops might have been able to scramble well enough to hit the deadline had we known about the project earlier.

Communicate

  1. The lack of a clear project plan or requirements list forced us to spend time gleaning that information from various sources prior to making progress on planning or execution.
  2. Incorrect communication to the customer about our ability to meet the deadline put us under the gun to deliver from the outset.

Preferred Introduction Process

Here is how I would expect a project of any weight to come across our organization’s radar. I am not saying that adhering this this particular process will remove resource “crunches”, late-binding requests, or competing high-priority projects. It will, however, allow for control over how projects are prioritized and how resources are distributed across them.

  1. During initial conversations regarding a new project, the PM and lead developer answer a base set of operational questions provided by the Ops team. The list of questions will vary depending on the request type.
  2. Answers to the questions are submitted to the Ops management team to be discussed internally. That discussion yields basic resource estimates and the assignment of an operational engineer to participate in design reviews.
  3. Shortly afterward, a project review meeting is held with all stakeholders to discuss requirements, resourcing, timing, and roles & responsibilities within the project team. The ops organization should send the engineer(s) slated to work on the project, at least one manager who can represent the org, and the PM responsible for the operational work.
  4. A full project plan should be published following the meeting, with clear requirements, deliverables and milestones defined. Project plans and meeting notes should be posted to an easily-accessible file share.
  5. The PM should review the plan with the customer quickly, and any requested amendments should be re-visited with the project team immediately.
  6. After the review of customer feedback, the project plan is locked down, and any new requests are considered “scope creep” and added to the project’s backlog in priority order. Each high-priority request should be discussed within the project team and should include stakeholders in order to determine whether it can be accommodated prior to committing to it.
  7. Regular succinct project updates should be furnished to all stakeholders to avoid last-minute escalations when a milestone is in danger of slipping.

Initial Project Meetings

The first meeting in any project should be a simple high-level walk-through of the project with whomever can define the proper resources for the initiative. This should happen as soon as the project has been approved and really shouldn’t last longer than 20 or 30 minutes. All we care about is getting the correct people in the room for the full scoping exercise and understanding the priority of the project compared to the other initiatives on the slate.

The second meeting should be of longer duration and include all of the resources necessary for delivery. I would much rather schedule a half day for this one than to head into a large project without all of the critical information. The agenda should cover defining the roles and responsibilities of each team member and ensuring that every participant understands the requirements well enough to scope their deliverables and provide reasonable time estimates for milestones. Engineers expect very clearly-defined roles and responsibilities up front, and I can’t blame them. Worrying about who they have to talk to about specific portions of a project, who makes what decisions, what bits of information are important and which aren’t, or who owns specific pieces of the technology direction or implementation just reduces their ability to do their jobs properly. Some questions to consider for that second meeting:

  • Who is the primary customer, and have they provided sufficient requirements?
  • What are the requirements and end goal of the project?
  • What is the priority of the project compared to other current initiatives? (while this should be addressed in the initial meetings, the engineers should also understand the relative priorities)
  • What is the communication channel with the customer?
  • Who is the subject matter expert for each major piece of technology?
  • Who makes decisions on the technical direction of the project? What happens if there is a stalemate on making a decision?

And the ‘soft’ stuff:

  • What is the escalation process? What is or isn’t worthy of escalation? Where does the buck stop?
  • What is the project manager’s role in the project?
  • What is the expectation around scoping dates for deliverables and milestones? Is contingency time added into each deliverable? Does the project itself have padding to allow for slips?
  • How often are the engineers expected to give an update on specific deliverables versus the overall progress toward the next milestone?
  • What do various stakeholders want to know about on a regular basis, and how would they like to see that information?

Answers to these questions should all be documented in an easily-accessible and visible place in the project’s doc repository from the beginning. Quite a bit of the information can be defined on an org- or company-wide level and shared across projects. I would expect a link to the document to be included in the header of the project plan and of every project meeting agenda. Any new member of the project should review the information with the PM prior to any participation in the project. Every manager who has an engineer involved in the project should also be familiar with the info, particularly anything to do with resourcing.

Regular project meetings should be scheduled (scrum/agile, waterfall, whatever). Every person with an outstanding deliverable, anyone expected to provide technical support and any “decision maker” for outstanding questions should attend or provide an update prior to the meeting if they’re unavailable. I’m a fan of stakeholders staying away from these meetings unless a prioritization discussion needs to happen. Too many pointy-haired bosses detracts from actual engineering progress.

the never-ending battle

Supporting Thoughts

Aside from the introduction of the project, there are many other facets of project management that are important for ensuring smooth delivery. Communication, defining roles and processes, and reporting at the right level are all integral to a project’s ongoing success.

Communication Is Most Important

The most important responsibility of a PM is to not only facilitate communication, but to have clear, crisp communication herself. This is important for the other participants as well, but engineers are typically paid to focus on technical delivery first and foremost.

Don’t be afraid of people. A PM shouldn’t shy away from human interaction (face-to-face, video conference, phone) and at least 50% of her day should be filled with talking to people (I pulled that number out of thin air, but it seems pretty reasonable). Like it or not, you can’t get the entire context of why a deliverable is late or why an engineer is spending “so much time” on a particular deliverable from a tracking ticket.

Minimize interruptions. I would also expect a PM to also strike a balance and be reasonable with the interruptions. If a deliverable is due in two weeks and yesterday’s update shows that it’s on track, I don’t see any reason to interrupt an engineer for yet-another-update. Everyone’s time is at a premium every day. Use discretion, common sense, and the guidelines that were set up in the initial meetings to determine whether that daily update is really necessary.

Be succinct. I don’t know too many people who enjoy reading a 3-page project update or listening to a 5-minute monologue, only to learn that the project is on track and there’s nothing to worry about. Verbal diarrhea during project review meetings is pretty terrible to sit through.

Call out important points explicitly in email. Use bold fonts, red colour or something similar to call attention to new actions items or issues. I’m a fan of the ol’ TL;DR format. A bulleted list containing what I honestly need to know about at that top of an email makes the team more efficient and appreciative.

Project Roles and Responsibilities

I believe that a Project Manager who is expected to lead an Ops project should have basic understanding of operational concepts. There are varied opinions on this, and some will say that if a PM asks the right questions, it doesn’t matter whether they understand systems or networking. I say that’s hogwash (because I’m old), and that if an Infrastructure PM doesn’t know enough about managing a system with cfengine/puppet/chef/etc, then they can’t guide the conversations, ask the right questions or help work out dependencies. At that point, they’re a Project Coordinator (big difference), and I shouldn’t be expecting them to drive a project on my team’s behalf.

PC/PM/PPM

I believe there’s a difference between a project coordinator, a project manager, and a program manager. Too often, these roles get confused or just plain aren’t defined, which leads to incorrect expectations from those involved in the project. Each organization needs to set their own job descriptions, but here are mine:

Project Coordinator: Project Coordinators are purely tactical. They get told what to do & how to do it, then it just magically gets handled. These are the people I prefer to work with. Part of it is a trust thing. I don’t dole out the responsibility for managing my teams’ resources easily, mostly due to the points I make later in this post. Part of it is that I’ve historically had strong technical leadership on my teams who can manage scoping and resourcing fairly well on their own.

[Technical] Project Manager: A really good project manager understands the resources and deliverables well enough to make suggestions about how to scope a project. They should also have enough knowledge to formulate considered, relevant questions to spur the right discussions. How does this project for X service impact its interaction with Y downstream service? I would not expect a TPM to know the answer, but I would expect a good one to be able to ask the question and grok the answer. I would also expect a solid TPM to call BS on time estimates that are just obviously way off base. The build-out of a new data centre is scoped at 3 days & you have no tools/automation to facilitate that? Dude.

[Technical] Program Manager: I confess to not having worked with too many honest-to-gosh program managers in my career, so I’ll just say that I would expect them to have a larger, more high-level view of really gnarly initiatives spanning multiple organizations, and to manage other project managers or coordinators who would take care of the lower-level responsibilities such as task tracking, updating project plans and keeping track of resources.

Other Project Team Roles

Engineer: Engineers should be focused on technical delivery as much as possible. There are a few other things that need to happen to facilitate that delivery.

  • Be realistic and take your time when asked for estimates on deliverables and milestones. Then pad those estimates, based on the project guidelines.
  • Know your audience. Keep updates less technical when necessary.
  • Learn about escalation – when, why, how. Escalation isn’t always a “bad thing”. Sometimes asking for assistance or feedback is the best thing you can do for yourself, the team and the project. If you’re already overloaded & don’t tell anyone about it, then we’ll continue to assume that you have the time to hit your deadlines or take on new work.
  • Being proactive on relaying progress, blockers and risks will enable the PM to remove those blockers and escalate issues before they become hair-on-fire scenarios
  • Provide feedback into the process constructively. Yes, Ops Engineers have that well-understood culture of…. curmudgeon-itis. Let it go, and be productive in your discussions with PMs. They honestly are trying to make your life easier.
  • Be patient: not every PM or customer has perfect technical understanding. If they did, they’d be your engineering team mate.

Manager: First point of escalation. In Operations, managers are typically better-poised to assign and manage resources across multiple projects. Project managers aren’t usually deep enough into the scope of our work load across all projects or the specialized skill sets within the team to address this themselves. We do include the PM in the conversations so that he understands what’s involved in the process.

  • Listen to the PM when he escalates to you. If something doesn’t make sense, then ask the relevant questions to get to the heart of the issue & then use the interaction to refine the escalation processes.
  • Hold engineers accountable for hitting their deliverables.
  • Listen to your engineer when she tells you that she’s overloaded, that a deliverable or milestone is in danger of slipping, or that the team flat-out missed something in scoping exercises. Not listening just sets the team up for either heroic efforts or missed deadlines.

“Decision Maker”: When there are conflicts regarding the direction or prioritization of a project, the buck stops with this person. For some projects, there may be multiple decision makers for technical issues or cross-project prioritization. You are responsible for ensuring you have all of the relevant information to make a considered decision. Make sure you are updated on the progress of the project and any potential conflicts. If you don’t have enough information to make a specific decision, then be vocal and ask as many questions as necessary to make the call quickly.

Define the Escalation Process

This could have been included in the section above, but to me, escalation is such a critical component of managing a major project that it deserves to be called out separately. While you’re defining the key decision makers in the initial meetings referenced above, you might as well define the escalation process, including details about what’s worthy of escalation and what isn’t, the method for escalating an issue, where the buck stops, and how to communicate that an issue has been escalated. For example, depending on your culture, it might not be acceptable to notify the entire project team if one person’s deliverable slips. A portion of this information could (should?) be defined at an org or even company level. Then all you have to do is play fill-in-the-blank with the owners for your specific project.

Defining this up front encourages people to communicate in the most sensible and productive way from the beginning, teaches participants to have open, frank conversations to try to work out issues on their own (no one likes being escalated on), and it saves the ‘punch’ of escalating for things that are legitimately critical. As a manager, there are few things worse than realizing that you’ve missed a critical escalation because it’s gotten lost in the noise of a million other unimportant pings.

Reporting

Stakeholder Updates

Each audience involved in a project requires different types and amounts of communication. I honestly didn’t understand this until I became a manager of managers and had multiple teams and projects to keep track of. As an engineer, I wanted to drink from the proverbial fire hose and have access to every tidbit of information possible. Nowadays, I manage four teams spread across approximately 15 projects at any given time. On average, I have somewhere between 10 and 15 minutes to devote to any one project per day.

Project Team Communication

The most valuable information project team members receive are updated project plans, progress against blockers they’ve raised, any changes to project scope, and changes to project timelines. I’m a fan of the daily scrum, where the team can all get on the same page in 15 minutes or less. It’s efficient when customized to your particular organization. Notes should be sent out after each meeting; simple bullet points should be sufficient.

Executive Summaries

There’s a difference between Executive Summaries and communication meant for a technical project team. If I’m asked to read and sift through a 3-page treatise in order to find the one embedded action item meant for myself or my team, it’s useless and a waste of my time. To boil it down, this is what I would expect:

  1. Is the project red/yellow/green based on the current milestone? If I see green, I need to be able to trust that things are on track and I can safely file it away.
  2. If the project is in jeopardy, what is the bulleted list of the blockers, who owns them, what the ECD (estimated completion date) is and what I actually need to worry about. I don’t want to see more than two sentences for any one bullet point. Do I need to find another resource? Do I need to step in and help guide things along?
  3. Did we complete a major milestone on time? Did one of my team members go above and beyond to improve the design, help the project team, etc?

General Status Reports

Include positives like hitting milestones. Make the communication clear enough and at the right technical level for the least-technical stakeholder who honestly needs to understand what’s going on with the project. C-level people and managers in organizations not involved in the project might be in that group (it all depends on your own org though, of course).

Basic Ops Questionnaire

It’s always a good idea to make it as simple as possible for PMs or developers to provide the right ops-related information for an internally- or externally-driven project. If you don’t have a formal NPI (new project initiative) process where these questions are codified, a simple document should suffice. We’ve begun using a google form open to anyone within the company for these types of requests. Some of the questions we expect answers for include

  • What is the request? X customer would like us to host a few customer service tools in our data centre. We would like to extend Y service to serve global requests, rather than serving requests from just Z service.
  • Who are the main technical and project contacts?
  • Who will be administering the service? If said service is ‘hosted’ rather than an internal one.
  • Will you need authentication for your administrators?
  • Does any data need to be stored locally on the machines? If so, what types?
  • Is there an admin UI or set of management scripts to administer the app? Or just backend processing?
  • Is this a mission critical service? Do you require notification during scheduled/unscheduled events?
  • How will the service be monitored?
  • Do you need server/network redundancy? Honestly, I can’t imagine ever putting something into production that isn’t redundant, but some people just don’t want to pay for it!
  • Are there any specialized hardware requirements for computation, storage, etc?
  • Do you have capacity numbers through the next 12 months? If not, when do you expect to have them?
  • Is integration with other services internal or external to our infrastructure required?
  • What security considerations or concerns do you have, if any?
  • How will you handle code deployments?
  • How will you handle packaging?
  • When do you need the environment(s) in place?
  • Are you prepared to furnish operational run books?

This is by no means an exhaustive list, but it helps drive the proper conversations from the beginning, and it also helps Ops management determine which resources to devote to a particular project.

Advertisements