Eleanor Saitta Eleanor Saitta

How to Organize an Engineering Team So Security Work Happens

Part One: Decision-Marking and Process

Once you’ve decided that it’s time to start a security program and you’ve figured out what work you’d like to start with — either from your own expertise or an outside review — it’s time to figure out how to get the work done.  For some companies, this is easy.  If the technical founder or CTO is security aware, comfortable balancing security risk versus product and market needs, and the team is small enough that they can manage everyone’s work streams, that’s pretty much that.  We’ve done security reviews for a few clients in this sort of situation, and they’ve gotten great outcomes.  Most companies, though, run into some complications when they try to put a security program into practice.  In this essay and its sequel (coming soon), we’ll look at the key features an organization needs to successfully roll out and execute a security program.  In this piece, we’ll focus on executive decisionmaking and the project management process.  In the second part, we’ll look at what’s needed in the company’s technical ecosystem and engineering team structure, the different categories of security-related work, and how they’re best staffed and organized.  Throughout this, we’ll talk about the CISO as the security decision-maker; think of this as a placeholder for whoever is doing that work, as long as it’s someone who isn’t the CEO or CTO.  In part two we’ll also talk about the different shapes that role can take and what makes sense when.

Executive Decision-Making

The way the executive team makes decisions is the single largest make or break issue for security program success.  If it’s aligned with the fundamentals of how security works for organizations, things are easy, but if it’s not, nothing else that happens in the rest of the organization will save the security program.  It may still look good on paper, but if, or when, it’s put to the test for real, the technical outcomes will not be good.  And it’s not just security that can be impacted here — you may see an impact across the entire business from a security program that’s badly run at the executive level.  We can break what’s needed down into five key components:

Trust

The first and most important requirement is trust, both toward the other members of the team, including the CISO, and self-confidence on the part of the executive team.  Few CEOs and COOs arrive at the point of starting a security program already understanding the ins and outs of security, even if they have some technical background; even many CTOs don’t.  This means they’re going to be spending a lot of time making consequential decisions about subject matter that’s a little foreign to them.  Good explanations always help, but trust is the core of it.

One of the most important skills for an executive is being comfortable with not knowing things — and with other people knowing that they don’t know something.  This takes confidence, but it’s the starting point of all decision-making processes.  If you aren’t comfortable with what you don’t know, then working with experts to fix that gap is going to be hard.  During the process of starting a security program, you’ll learn a lot, but you’re still going to be both leaning on explanations from other people and making decisions with partial knowledge.  If you trust those people and you’re comfortable in this process, things can go smoothly.  On the other hand, if you don’t trust the people giving you technical and security advice, the process is all-but-guaranteed to fail.  Find a team that you trust.1

One of the best ways to build trust is to treat the CISO as a full member of the executive team.  Many companies have the CISO report up through the CTO, and we see this as a mistake, as it keeps the CISO from understanding the full context of the business and from having a deep relationship with the rest of the executive team.  As the company grows, this becomes less important, but in small companies, the CTO is often product-focused.  To be clear, this is good!  It’s critical for a company that’s finding product-market fit.  As companies get larger, the role of the CTO moves toward shepherding the entire engineering organization, and it’s easier to take on a broader perspective.  In our experience, most firms under 200 technical staff are better having the CISO as a peer to the CTO.

Market and Threat Awareness

So, how much security is enough?  Every CEO or CTO has some version of this question, and the answer is never as clear as anyone would like.  A lot of things feed into this — what level of security do potential customers in your industry demand?  How regulated is your industry, and what do the regulators demand?  How targeted is the company’s line of business — are you just dealing with “being a company on the internet” risks, or does your sector have specific risks?  What is your tech stack like — have you made it easy on yourself, or are you in a position where getting to an appropriate level of security is going to take much more work?  Finally, who’s after you?  You may be just running a chain of fast food restaurants, but if you’re doing it in Kyiv today, you have a different risk profile than you do if you’re in London.

As a CEO or CTO, you should have some understanding of what your customers and regulators want, at least, but the rest of this may be new to you.  Getting to a shared highlevel understanding on market demands and security threats that everyone on the executive team is confident in is critical, as is understanding that new information can change the rest of your security program and impact products too.  We’ve had multiple clients who started out selling to mid-sized companies and had a real shock with their first few big enterprise clients.  If a shock like that happens to you, take advantage of it — the world has just delivered you some expensive information and the sooner you learn from it the better.  It’s not unreasonable for the executive team to want some additional information from sources other than their CISO — talking to their peers at other companies in the same industry, for instance, can make sense.  At the end of the day, though, every company is different.  There may be factors that put your firm in a different place than peers that seem similar to you.  There’s a fine line between information gathering and second guessing or trying to negotiate with reality, and this is where trust is critical.

How deep of an understanding of market demands, threats, and security strategy each member of the executive team will need depends on their exact portfolio.  In general, the CTO, CEO, COO, and General Counsel are most involved, but other folks should still have a general understanding.  Depending on the roles you have, more people may be on that list.  If you have a Chief Product Officer, for instance, they may need a deeper understanding than the CEO, even.  Talking through the structure of how work is split up should be an early part of framing the CISO role and their interactions with the rest of the executive team.

Strategy, Risk, and Trade-Offs

Once you understand what security looks like for your firm, the next step is figuring out a technical strategy for security and how that interacts with the technical and business strategy of the company as a whole.  This is where trade-offs need to be made, and the core of those trade-offs is the risk tolerance of the company.  If you’re in a heavily regulated or security conscious line of business, have solid product-market fit, and are on your way to delivering well without massive growth pressure, you may have a low tolerance for security risk.  Things are going your way and the only thing taking security risks is going to do is mess everything up.  On the other hand, if you have a high risk tolerance in other areas of the business, it may be the case that your security risk tolerance should match.

Security risk is complex, however, despite what many risk register vendors would like to tell you.  You’re not just taking a risk for the company as an entity, you’re also taking risks for your staff and your customers, personally.  Even the least risk-averse firm has some basic ethical (not to mention legal) obligations here.

Some companies also make trade-offs that come from the founders.  We’ve had clients with high bars for staff privacy, even with respect to automated tooling, who were willing to accept risks to the entire firm to maintain that privacy.  Similarly, we’ve had clients who valued independence from third-party vendors more strongly than SaaS-heavy industry norms, leading to technical trade-offs on security risk (not to mention product velocity).  It’s your firm, and you get to make those calls.  The critical issue is that everyone making relevant decisions understands the company’s risk tolerance and has a good rubric for the way the executive team expects decisions to be made.  If the folks doing the work don’t have a solid idea of what solutions will be acceptable, conflict is inevitable.

Most of the time, doing something for security means you can’t do something else with the same time or money.  Ideally, you don’t just have a long list of waiting security tickets, but a technical strategy for the way security outcomes will be achieved for the company that acts as a through-line across all the security work.  Explaining this strategy is on the CISO and CTO, and it’s worth talking through it until e.g. the CEO can recognize it when making relevant decisions.

For folks without a technical background the through-line of a security strategy can be hard to see, as in practice security can be kind of like trying to put your finger on all the holes in a colander.  That said, trying to negotiate with technical reality does not work.  There is a bar for security that comes just with being a company that exists on the internet, and that bar doesn’t change even you stare at it really hard.

If the CISO comes to the CEO and says they need two engineers for a quarter each for patching known vulnerabilities and system hardening, saying they can only have one doesn’t make the risks to the business from the other project go away.  If as a CEO you’ve expressed a risk tolerance for the company as a whole and the CISO is giving you a plan that achieves that risk tolerance, they’ve made the professional judgement call you’re asking them to make.  If there’s context that makes it impossible, they should have it — and then work with you to get to a plan that is possible.  The right time to handle this is up front, making sure that as the CISO is starting to develop the company’s security strategy, they have a clear understanding of what resources are available, what the constraints and goto-market or product strategy is, and even what investor relationships are like.  It’s possible to hold all this information close to your chest, but then you can’t complain when you get a strategy based on what the CISO does know.  Reaching out to your board to help understand these risk trade-offs can be useful, and including the CISO in that conversation is helpful.  Remember also that as a CEO, you get to frame this conversation — if you choose an us versus them framing between security and product, you will get the result you asked for.

In reality, these trade-offs often aren’t zero sum.  A good CISO and CTO should look at how the security strategy interacts with the technical and business strategies of the company.  In some cases, the work needed for security can open up possibilities for the whole company’s technical strategy, and in the best case even create business opportunities that weren’t possible before.  This kind of strategic synergy2 requires that the CEO, CTO, and CISO all understand the high-level technical and business strategy, including the pressures, market demands, and structural options that the business may be exploring or facing.  The business side of this is sometimes seen as a long way away from the CISO’s domain, especially in less technology-centric companies.  While the CISO doesn’t need to hear every detail of the sales team’s current challenges, they should have access to as much of that information as is practical.  A good CISO is a deep partner of the CTO and their work will impact the business as a whole.  As before, trust is key.

Commitment and Decisiveness

Some executive teams are good at making decisions and sticking to them, even to a fault.  Some aren’t.  A security program at a company that hasn’t had one before is almost without exception the largest set of work the company has undertaken that’s not driven by product-market fit, if we exclude non-security regulatory demands in highly-regulated industries.  It will impact the rest of your roadmap, and it will impact your ability to implement last-minute product changes.  This can be terrifying, and for some executives, it means making a kind of decision they haven’t faced before.  Making the wrong decision and sticking with it until you get information that demonstrates that you were wrong is often less bad than being indecisive.  Now, if the wrong decision was made, the company still has to weather the event that reveals this — in some cases, hindsight might show a much brighter alternative world.  When it happens, though, the only thing to do is replan and move forward.  Failing to make a decision, or worse, getting into a loop of starting, getting cold feet, doing something else, and then getting corrected back to the original plan can be far worse than just sticking to the wrong decision.  The cost of security work is hard to estimate ahead of time (more on this below), but stopping and starting or reprioritizing is a sure-fire way to make it more expensive.

Cultural Leadership

As goes the executive team, so goes the company.  If the executive team expects everyone else to comply with a security policy but lets themselves be the exception, everyone else will try to get out of it — don’t be that guy.  Starting a security program is often one of the first changes from being a startup to being a big-boy company, and one of the failure modes is when security end up being the fun police who no one wants to work with.  While there aresome things that a security team can do to mitigate this, the executives are the ones who set the tone here.  If they make clear how things are going to change — and here’s a place where it’s useful that the whole executive team understands how security work will impact people’s work, especially on the IT side — and why it matters for the company, showing that the security team has their trust and approval, outcomes are better for the whole firm.  This is why making it clear that they’re following the same rules as everyone else is so critical.  It’s a big transition for the whole company, and it’s a lot easier if everyone is on the same side going through it.  IT and security can handle all the day to day communications, but they need the CEO to make it clear that the changes are coming from the top.

As an aside, there are two choices you can make early on as a founder that will make your life easier when it’s time to start taking security seriously, especially if you know the company will have a low risk-tolerance.  First, set norms that company devices and accounts are for work, and that while folks can do some personal stuff on them, they shouldn’t think about these as personal devices.  Partially, this is just a good idea overall — anyone who has ever dealt with the discovery process around a lawsuit against their employer is happy to keep everything in their personal life as far away from work devices as possible.  Primarily, it means it’s less of a shock to the company culture when devices start getting hardened or monitored more closely.  The other choice is to be very, very, very careful about never using the access you have to company systems to check up on employees except in cases where you’re trying to rule out malicious action3 — and even then, it’s best to be up front about it to the rest of the team, even if it’s after the fact.  Building a security program unavoidably means that the security team will have access to tools and data that could violate the privacy of employees.  If staff can’t trust that that access will be used with the utmost care and won’t be used for any purpose other than technical security outcomes, there will be much more conflict around a security program rollout.  If you’re worried about one of your employees slacking off or working their side-hustle on company time, do not use Workspace SuperAdmin access to catch them.  Go have the uncomfortable conversation instead.

Process

Once the executive team is onboard with the work, knows how to talk to each other and make hard decisions, understands the risks and threats, and is showing the company that they care about security, it’s time to get the rest of the company moving.  In small companies, the CTO says go and both developers get to work and the whole problem fits in everyone’s head — but teams this small are going to be hard-pressed to start anything that one would call a security program.  The team size where starting a security program makes sense is around the 10-20 engineer mark (more on this in part two), meaning there are at least two different streams of work happening, and maybe more like a dozen.  Security is often the first stakeholder needing to get engineering work done that’s not driven by the product roadmap or vision.  Individual engineers or teams may have had internal work that they’ve done alongside their product-driven work — tooling they needed, quality of life improvements, bug fixes, etc. — but this is driven bottom up; they’re the customer for their work.  Now that there’s a second top-down stakeholder, the way decisions are made needs to change.

Clear Planning Decisions

The process by which the team makes decisions about what work gets done does not matter as long as a) it’s not too onerous on the team and b) both the CISO and CTO understand what the decisions are, have a say in the decisions line with the priorities set by the executive team, know how the resulting work is going as it happens, and are both involved when the work gets off track or priorities need to be changed.  Above all else, how and when decisions are made and what decisions have been made needs to be clear to everyone involved.  This is more difficult than it sounds.

The first requirement in planning work is that there be a plan.  It’s not uncommon to see early teams where there is no plan beyond what folks will start working on next Monday.  Sometimes this is because design and development is happening just-in-time, in response to customer requests day by day.  Sometimes there’s a grand plan in someone’s head and there hasn’t been a need to write it all out.  Starting a security program is often a driver in moving from a sprint planning plus vague notes in a ticket backlog to at least limited quarter-scale planning.  Once a plan exists somewhere, there’s a place for the CISO and CTO to write down the decisions they make and a way for the rest of the team to understand those choices and the priority they have.  Security work is often the definition of important but not urgent (until it would have needed to have been done last week), and without a plan, urgent product work can keep cutting in line.

Once there’s some kind of plan (and this can be a whiteboard or a text file in source control), there needs to be a way that everyone agrees on for how decisions get made.  It’s also necessary that decisions are made4.  If the CTO, their two managers, and the two most senior engineers get together every Monday and make those decisions, it’s easy — make sure the CISO is there too, call the group back together if the plan changes on Wednesday, and you’re solid.

In some cases, especially in fast moving teams weighted toward senior- or staff-level engineers, the CTO may be steering by expressing some general intentions and then expecting their team to go make the actual decisions and do sensible things.  This is great, until someone else comes along and also expresses some general — or worse, specific — intentions.  If the team is used to executing without checking back in — say there are 25 engineers and the first manager starts next month and the CTO is out of bandwidth5 — this can result in surprises.  Even if the end result is good, the surprise itself causes problems.  An organization like this isn’t going to spin on a dime to epics and Gantt charts, but they don’t need to.  The CTO, with input from the CISO about their needs, needs to make it clear to the team what kinds of things they have the authority to make calls on, what they want to be notified about, and what decisions they need to be involved in.  They also need to provide a channel where the team can get decisions back in a timely manner, and make sure the CISO has visibility too.6

Whatever structure is used, it needs to apply to everyone.  If the CEO gets a critical request from the biggest client, it needs to go through the same process — even, and maybe especially — if there’s no way that won’t be the top priority.  Even if all the other plans get swept off the table (it happens, and it’s often the right call), everyone knows what happened and why.  If there’s a process, it needs to be the process.  If the process is too heavyweight to follow in a case like this, it’s the wrong process for the company at this stage.  See if there’s a lighter, faster version that still does what’s needed.

In larger development teams, we’ve seen issues where there’s a plan on paper, but it doesn’t relate much to what happens.  For instance, if a team has a big product project and a big security project and both of them are supposed to get done this quarter, but the product project has their quarterly bonus riding on it, the security work isn’t getting done.  You get the work you incentivize, and if you’re going to try to use incentives to drive team performance, they should map to the priorities you agree on.  This problem gets worse when it’s not clear to all stakeholders what work is even happening.  If you end up with a decision process that only has one cycle per quarter at the CTO/CISO level for prioritizing work and work isn’t done to plan… well, good luck.

Estimation

If you’ve got two pieces of work that are at an equal priority, it would be useful to know how long they’ll take when you’re deciding what to work on.  Unfortunately, guessing ahead of time how long it will take to get a nontrivial piece of software into a specific state is a fool’s errand7.  Instead of deciding ahead of time that an unknowable amount of work will take 29 and ¼ days, ask the team that owns the system in question what kind of a job they could do on a given feature in a given amount of time, and what the contingencies look like.  Work will expand to fill the time available, but it will sometimes also contract.  For instance, you might be able to replace the vulnerable pattern that you’ve been using for some set of operations with a safer library so new development can go forward in a better state and then pick five random legacy instances to fix in the time you have, but you can’t also fix all of the other places the old pattern is used.  Fine!  Now you have a better instinct for what doing the rest of it will look like, you’re no longer adding to the technical debt pile, and the next time this work comes back up the priority stack, you can get further.

Now, as a security person, this pains me — an incomplete migration does mean more tech debt because there are two ways of doing things, and parts of the system are still vulnerable.  However, this is better for me than either never getting that work scheduled because I can’t get exclusive use of that team for an entire quarter at once, or starting the work with an estimated two weeks scheduled and then hard blocking two other teams when they run a month over — meaning I’m definitely not getting their time back to finish the job later.

Early in the team’s security journey, time-boxing work so you can get some improvement in a bunch of different places while doing other work is often more useful than trying to get things perfect in one place.  There will always be more security work that could be done than can be done, even if you do nothing else.  That said, sometimes you will have specific fixed security requirements, the same as any other product requirement.  Most often this is either a direct compliance requirement, or a compliance requirement that your customers face.  Here, it’s important that you treat these as what they are — requirements for your product to find market fit.  Yes, your security team is going to be taking the lead on converting what you’re hearing from your customers into specifications, but it’s important to distinguish between work the security team selects to reduce the risk to the company and work the company does so they can sell to customers (even if it will also reduce risk to the company).

Interleaving Work

There is a cost to make a change to a given part of a system — any change at all.  An engineer may need to remember how that service gets deployed, relearn the structure of the code, or go familiarize themselves with a piece of the tech stack they haven’t touched before.  If you’re going to pay that price for one thing, you might as well get two things done at the same time.  If a system is getting replaced, adding some additional architectural changes beyond the one that drove the upgrade is often cheaper.  While you’re in there reworking how we do all the billing tracking, how about we also stop passing all the account data around in big JSON blobs so we don’t need to parse risky customer-supplied data everywhere.  This kind of interleaving of work is a big part of making security changes cheaper when you have legacy systems, and finding opportunities for it is key to a cost effective security program.  This is why the CISO doesn’t just need to know the plan for security work and how it’s going, but also everything else happening in engineering.  The whole team (in small organizations) should know the plan too, so they can help spot these chances.

End, Part One

In this part, we’ve gone over what you can do to make sure your new security program will be successful within the executive team and in terms of how you plan and decide on work.  In the next part, we’ll talk about the kind of resource limits you may run into when building out a security program, the nature of the five different kinds of security-related work, and how best to take care of each one.

Systems Structure Limited is a boutique security consultancy, providing fractional CISO services to startups.  Over the past nine years, we’ve built security programs for fifteen different companies, and our work has been responsible for hundreds of millions of dollars in capital raised by our clients.  If you’re reading this essay because you’re getting ready to start a security program at your startup, let’s talk!




Footnotes
  1. Getting comfortable with not knowing things and self-confidence in general is outside of the scope of this essay, but therapy is a great start.  Just saying.
  2. Ugh, I can’t believe I just typed that, but that is literally what we’re talking about here.
  3. If you’re tempted to do this, talk to your lawyer first and tell them I sent you.  They’ll thank me.
  4. You would think this was obvious, wouldn’t you?  But a lot of teams have issues here.
  5. I have never met a CTO who didn’t have too much on their plate.
  6. Even if the CTO does have bandwidth and there is a planning process, talking through with the team where the limits and expectations of agency are for them will pay dividends, especially in emergencies.
  7. Many people will tell you differently, especially before it turns out they were wrong.  These people are rarely the ones doing the work
Read More