How to Organize an Engineering Team So Security Work Happens: Part Two

Sep 23

Resources and Work

This is the second part of two part essay on how to get security work done when you’re starting a new security program done at a company. In the first part, which you can read here, we focused on executive decision-making and the process of managing technical work. In this part, we’re looking at the properties of the technical ecosystem and the team that each have the biggest impact on security program outcomes, the different categories of security work, and how they’re best staffed and organized. First, though, a word on “capacity”.

Capacity

This is the thing that CTOs tend to worry about when looking at starting a security program — do they have the resources, both in dollars and engineer-hours, to do the work that this entails. Sometimes, the answer is no. We encourage teams to not start too early — if you don’t have product-market fit, you should be thinking about the security you need to be an ethical guardian of the data, money, and trust of your employees and early customers and to meet your legal obligations, but not more.¹ Every company is different, but a good starting point is to wait until you’re around the ten engineer mark, but no later than the thirty engineer mark (and preferably twenty). You will need at least one person on the team dedicated to infrastructure (out of ten; more if you’re bigger), and it’s great if someone else can be at least 50% on IT work for laptops and the like. Budget-wise, while you can spend much more, expect a minimum of about $200/employee/month across both IT and infrastructure/development tooling. These are ballpark numbers and will vary by industry, technical stack, and strategy, among other things.

That said, capacity isn’t always the bottleneck. Yes, security tools get expensive and the work can eat a lot of time. You can trade time for money and vice versa, but only so much — more on this later. You can also hire or fundraise your way out of a capacity problem, but this won't necessarily fix the constraints on getting security work done.

Mutability

At least two thirds of the initial work we put on clients’ roadmaps is basic engineering work, getting their production environment, CI/CD, development practices, and IT management up to par. That can be quite a lift, but it pays the team back with more reliable systems and, usually, an environment that’s nicer and faster to work in. The biggest resource the team needs for a successful security program is the ability to change their systems. The clients we’ve had who have struggled the most with getting security work done have had legacy systems they’ve inherited from an earlier stage in the company’s life or critical components built by people who left no documentation. Your fancy all-Terraform and Control Tower AWS organization doesn’t help if your legacy account has all the critical data in it and you’re 40% into a migration effort that’s already taken two years of work from 20% of your engineering team.² Adding people to a late project famously doesn’t make it faster and may make it slower.

The bigger the system, the harder it is to understand and secure. Think about every line of code you write as a liability, not an asset — it needs to be patched, maintained, refactored, monitored, and responded to. You spend code to buy features. Code quantity limits system mutability and thus system securability. Keeping your system as simple as possible to solve the problems you actually have is key. Why are you rolling out a clustered or sharded database when you haven’t profiled your database queries? Is your 99th percentile database load really too big to fit with room to spare on a single AWS db.m9g.48xlarge instance with almost a terabyte of ram across 192 vCPUs? Almost certainly not, and yet by clustering you’ve bought a system that will be more fragile and complex and will be harder and slower to change.

Technical debt adds complexity just as fast as early optimization and scale. If there are two ways to do something because you never finished a migration, you have to check both paths for vulnerabilities and make sure they keep working. Keep things as simple as you can for as long as you can and finish every migration you start.

Slack

The next resource CTOs should worry about is slack — how much free time your technical staff have. If your answer is “none”, you’re accumulating technical debt in ways that you don’t understand and likely have a growing list of staff who are single points of failure. If you have a serious, sustained emergency — and eventually you will — the recovery process is likely to burn people out. When they quit, they’ll take irreplaceable system knowledge out the door.

Slack is linked to system mutability and the capacity to do security-focused work. Every process of system development and maintenance will have unplanned work. Developers will notice code that smells bad or come across weird behavior unrelated to their current work. If they have time to go back after they finish that ticket, great. If they don’t, these things add up and make the code base fragile and harder to change. Other things that often end up in this category (but shouldn’t) include writing new tests, documenting behavior, and teaching other engineers about parts of the system they understand. In a healthy development organization, this will be between ten and twenty percent of an engineer’s time.³ This kind of work is also key to the feelings of pride and ownership that prevent burnout. If engineers can’t do this, and worse if they’re working overtime or not taking vacations, emergencies are more likely and will be harder to recover from.

Security is, unfortunately, a source of emergencies. Making them less likely is the core of the work, but emergency patches to upstream vulnerabilities and incident response are unavoidable facts of life. Part of building an effective security program is making sure the team has the capacity and resilience to absorb predictable emergencies without harm. As an executive team, you need to understand that this overhead is necessary and resist the temptation to overload the engineering team with product work. Between slack, meeting time, and regular security work, it’s unlikely that a development team can sustainably spend more than 80% of their working hours on product work, and even 70% is usually aspirational.

Slack is not a capacity issue. Adding more engineers in the same culture with the same incentives to their line and project managers and the executive team does not mean more slack, it means more churn on the product. Sometimes, especially pre-product/market fit, this does mean success for the company — that’s often where bad habits start. Engineering quality and resilience are hard to measure and their impact on the bottom line is slow. Even if you build a culture that values them across the engineering team, if that change doesn’t reach the rest of the company — especially product, sales, and marketing — it will be eroded. Enforcing rules about slack at the CTO level can help, but nothing beats culture in the end.

Now that we’ve talked about the less-expected resources, let’s talk about what security work itself is. There are five categories of scheduled security-related work:

Detection — finding and triaging potential security incidents⁴
Security-specific infrastructure work — building tools needed for other security work
Hardening, patching, and architecture changes — work on production or IT systems done for security purposes
Security testing and analysis — penetration tests, code review, etc.
Security governance — managing the overall security program and processed

Detection and Security-Specific Infrastructure

The first two categories have available time/money trade-offs that most small companies should take advantage of. Especially as the speed of exploitation is increasing, you should have 24×7 coverage for detection if you have online infrastructure. The smallest sustainable team for that is six people. While you need the coverage, companies with less than a few hundred engineers do not need this much in-house capacity unless there are industry-specific extenuating circumstances. Yes, these folks can do other security-related work on the side, but being good at detection and investigation is a different skill set than being good at developer enablement, compliance, building infrastructure, or any of the other relevant tasks. Good generalists are more expensive, harder to find, and in more demand. Most of them also don’t want to do shift work on detection. It’s more effective to outsource that work to teams of people who do nothing else and are happy about that. Detection also isn’t just a matter of having the bodies and a SIEM. Building out intelligence channels, response and review automation, tracking, processes, etc. means you need senior talent, not just a pile of juniors, and can take a year or two of work across the whole team.

Many, many companies would now like to sell you AI-driven solutions that they claim eliminate the need for humans in the detection process, but in reality, they at best reduce it. There are some kinds of anomaly detection that are difficult without flexible automation, especially when done in real time, and agentic or LLM-based systems can do more than conventional automation in some ways, but also cause false positives that are more plausible, are often no better at understanding the human context of actions, and also add additional failure modes. A human in the loop in the detection process, especially a smart human with the experience to catch subtle things, remains indispensable. Among other things, when you have orchestrated remediation actions, the human can keep your detection and response tool from itself being used as an attack vector.

As with detection, security-specific infrastructure is also better consumed as a service, especially at small companies. A reputable SIEM vendor is going to be better at both running their service and securing the ecosystem around it than most small to medium companies will be. They also exist outside of your internal trust boundaries — an attack that compromises your systems is unlikely to also compromise their systems. This means that an attacker is less likely to be able to delay detection by suppressing alerts that might tip you off and you’re less likely to be flying blind during the incident response process as the attacker can’t delete logs.

Building additional pieces of critical infrastructure internally is expensive if you’re not already in the building lots of critical infrastructure business, which few startups are. When you’re pricing SaaS tools versus in-house hosting, you need to get a realistic understanding of the work required to maintain the tool and price in the cost of the folks who will manage it, the people who will manage them, etc. Talk to people who have seen the tools you’re looking at go wrong, not just the happy path deployments. A single Elastic Search cluster that goes wrong enough at scale can burn out an entire team.⁵

If you do decide to build security infrastructure in house, it should be built by your regular infrastructure team, not by the security team. Security infrastructure needs to work in the same way and follow the same rules as everything else in your ecosystem, including the deployment process, run books, documentation, and anything else common across all of your systems. Security engineers are not infrastructure engineers, even though they may have a lot of common skill sets. Security teams are incentivized to build infrastructure in a hurry and cut corners so they can get back to their real work, and can even be the worst offenders on things like patching. For work scheduling reasons, you may want to take part of your infrastructure team and second them to the security team to get infrastructure built, but folks there should be rotated through so everyone gets to know the security systems, and that team should maintain the same standards and practices they have in their normal work. If you have a mature engineering culture, none of this may matter, but this is rare in small companies. If you have a sensitive system that needs to be compartmentalized away from the rest of the engineering team, it should still be built in this way but handed over to the security team once it’s in place (with sufficient documentation for them to maintain it independently). If your security strategy depends on the rest of the engineering team not understanding the architecture of a security system, your strategy is wrong and you need to start over.

Hardening, Patching, and Architecture Changes

This category is core product work, the stuff that we talked about scheduling in the process section of part one. When you start your security program, there will be a long list of this work. It’s not uncommon to come out of a whole-organization initial security review with twenty or thirty epics of work that would take one or more years for the entire company to work through. Now, obviously the team is not going to do nothing but security work for a year. Much of this initial work will be general engineering work, getting the system into a state where changes can be made quickly, paying down technical debt, and getting things to a place where the system can be secured. With that caveat, it’s likely that security-driven work will take between a third and a fifth of team bandwidth for the first two or three years of a security program. In some cases, it has been more efficient, especially for infrastructure teams, to have one or more quarter-long security pushes where they get larger pieces of work done more. However, in most cases the key is tight coordination with product planning so security work can be streamlined in with product work that touches the same systems. The point at which the company can afford to spend time on engineering quality is also when a security program should be started — you can do it earlier, but you shouldn’t wait longer. There’s a lot more to be said about the details of this work, but it’s more about the technical nature of the work than how to get it done, so that will wait for a later essay.

Architectural changes, at the level of the team doing the work, are just like any large development project, but security input to architecture merits a specific note. Good security architecture decisions need someone who has seen a large variety of systems and who can work at the systems design level. A surprising number of even staff and senior staff level engineers are, surprisingly, not systems thinkers as such — they’re just good developers. Folks in this category are often decent at building systems that handle the type of requirements they’ve dealt with before, but less good at synthesizing architectures around new types of requirements. Companies doing novel technical work with significant security exposure should hire in the architectural talent they need in-house, as this will be a significant part of the company’s core business. For small teams building traditional systems where security is a primary driver of architectural change, bringing in a CISO who can also do architectural work can be a better balance, whether they’re full-time or fractional; more on this when we talk about governance below. If you do not already have a process for how architectural decisions are made across the engineering organization, or the process you have doesn't provide a way for people outside the team doing the work to give feedback, it's time to change that. Some variation of architecture decision records are often a good fit, with the caveat that there should be somewhere central (Slack is fine) where people post when they're starting the ADR process. This ensures that folks from the security side can get involved early if they need to, rather than needing to be the fun police overruling a decision everyone else is already happy with.

One thing that companies sometimes do to handle the tension between the product roadmap and the security roadmap is to make the security team responsible for fixing security issues in production. This is always a bad idea. The team responsible for a system is the one who knows it, and having someone else come in and start changing that system is inefficient (they don’t know the system, so they’re slower), dangerous (the team that owns the system now doesn’t know some parts of it), and unreliable (work conflicts between the security team and the owning team can make the whole thing take longer than if the owning team just did the work). Whoever owns the system needs to be the one fixing it. A corollary to this is that every system must be owned by some team, and that team must have the bandwidth to fix that system, even if they’re doing development work on other systems. Failing this means you cannot change your systems and thus cannot secure them.

Security Testing and Analysis

This category of work covers a few different areas — penetration testing, application security testing, security code review, and cryptographic analysis,. Different aspects of this are handled differently, by a mix of tools, short-term consulting, long-term advisory support, and direct hires. In small firms, most or all of this work will be outsourced or handled by tools. As the company grows larger and matures, some of these functions will be brought in-house. In some cases, compliance regimes or market requirements mean that work must be outsourced — PCI, ISO27001, and SOC2 audits are always external, for instance, but themselves also require external penetration tests.⁶ Even without requirements for external review, small companies don’t have enough of this work to make sense to do it in house. It’s unusual to see folks focused on penetration testing on a security team smaller than a dozen people.

Application security review and security assistance to developers is the first set of testing work that should be brought in-house, especially for teams that write significant amounts of high-risk code.⁷ It’s often difficult to find application security engineers focused on developer enablement rather than just finding bugs and tossing them over the wall. We have had luck work with taking security-interested developers, either who already work at the company or new hires, and teaching them the security testing part of the job. The goal of this job is helping other engineers get better at finding and fixing bugs themselves and steering development practices toward safer patterns and libraries, along with the occasional code audit. This engineer may end up writing some code — things like high-risk data validation libraries are a good example — and will fix some bugs as a way of demonstrating the pattern for the fix to others. However, it’s critical for the sustainability of a security program that engineers do their own bug fixes. Secure development training is useful, but secure development practice is much more instructive.

A single application security engineer can’t be everywhere, so tools fill in the gap. Static and dynamic security testing tools will catch many bug classes, especially when combined with fuzzing. Bringing SAST in early will pay dividends, as developers find bugs early rather than replicating them throughout a code base where fixes will take far longer. AI-assisted fuzzing is also showing good results, even in complex code bases. Despite the hype, other types of AI testing have a smaller improvement in practice — about twice as many true positives and half as many false positives is a good rule of thumb, but the human review process often takes longer than it did with conventional tools.⁸ LLM-based testing is also often no better at things like logic issues in business code that require understanding the human intent of the system. External human review remains the gold standard for security-critical code. Two things that may not be obvious to folks who haven’t both written code and done application security review professionally are that reviewing code for errors is harder than writing it and that being good at writing code doesn’t make you good at reviewing it. For high-risk systems, it’s best if someone who does only code security review takes a swing at it.

Unless you have a staff cryptographer, meaning someone who has a doctorate in the subject, cryptographic reviews will always be outsourced to someone who does. If you're just implementing an existing, well-documented cryptographic protocol, it should still must get external code review, but if you're implementing a novel cryptographic protocol or an existing cryptographic primitive, it needs both code and cryptographic review. If you are implementing a novel cryptographic primitive in commercial code, you have without exception made a terrible decision and need to re-architect your system so this is not required.

Governance

Governance work covers a lot of different areas. At the top level, this means determining the company’s threat landscape, setting company strategy around security, communicating these and the company’s current level of exposure to the rest of the executive team and the board, handling the risk acceptance process, and ensuring that security work is adequately resourced. At the operational level, this includes everything from process building and policy writing, vendor security reviews, decisions on risk, and compliance tracking to managing external security audits, scheduling and tracking security-driven technical work, and communication about security with customers, regulators, and vulnerability reporters. Some technical work is also effectively governance work — making decisions about security architecture and running incident response processes. There’s also the job of working with the CTO to make sure security work gets done — helping to set up planning processes if they don’t exist, advising on team structure and talent gaps, and negotiating joint policies and processes like the SDLC and code review.

In a large company, while the executive and CTO liaison work stays with the CISO, the rest of this work (along with some of the previous categories) is split across a security department. On small teams this is often all on the CISO. However, some parts of this work can be split off at different points even in smaller teams. If the engineering team has a project manager, they can handle the tracking of security work alongside the rest of the engineering work. If there is a dedicated IT staffer, they can handle the first pass of vendor reviews and data gathering and pass the results off to the CISO for the final sign-off.

Compliance regimes can also be split off, but this requires care. If a small company is going through ISO 27001 or SOC2, there are three roles that need to be filled. The first is the named security officer responsible for executing the security program, as per the standards. This will always be the CISO. The second is the person who is gathering the compliance evidence to show to auditors. This is time-consuming work that requires a good deal of process compliance, but not seniority or — as long as there’s an easy escalation path to the CISO — much technical security experience. The third role combines, before certification, advisory work as to the requirements for certification, policy completeness, etc., and, immediately before and ongoing after certification, internal auditing prior to each external audit. The evidence gathering and internal audit roles are frequently combined into a single compliance manager position, but this carries risks.

Compliance-first security can have a serious impact on product velocity, company culture, and staff morale, and usually doesn't result in secure technical outcomes. It’s important that the CISO, not compliance, remains responsible for security strategy and architecture decisions that serve the technical and business needs of the company, including when to push back against narrow compliance mandates that are harmful or incoherent in production systems as they exist. If compliance runs strategy and no one else on the team has a security background, it’s easy for a useless checklist-driven approach to look effective until there’s a major incident. If there is a compliance manager, it’s best that they report up to the CISO, rather than being a peer. This ensures that the goal is security and that compliance is a tool to get there, rather than the other way around. Alternately, keep evidence gathering in-house, possibly run by the engineering project manager until that becomes too much workload, and outsource the compliance guidance and external audit function.

The broad nature of governance work presents a hard problem. When a company has between about thirty and a hundred engineers, combining all the strands of work mentioned at the start of this section results in just enough work for one full-time role. Unfortunately, most folks who have the background and seniority to handle that scope of work well and who want an in-house job want to be running a team. Folks who don’t want to be running a team are usually missing some crucial skill areas — either they’re great engineers but don’t understand or want to do governance or policy work, or they understand governance but don’t have a technical understanding of how security works.⁹ There are unicorns here, but they tend to know their worth. Just because someone who could command a seven figure salary as a CISO wants an individual contributor role within a smaller team doesn’t make them that much cheaper.

This dilemma is why Systems Structure Ltd. exists as a consultancy — we saw a lot of small teams that needed our depth of experience, but none of them had enough work to justify a full-time role, especially when there are easy options to shift the parts of the role that don’t need that level of experience away. If a firm has fewer than a hundred engineers, they are generally better off with a long-term advisor filling the CISO role than bringing the role in-house.¹⁰ This is true even if they have security requirements that drive a higher degree of security complexity or assurance or more regulatory burdens, but the threshold may be lower in those cases — somewhere around the fifty to seventy engineer range.

Growth speed and regulatory burden both impact this. If you have a forty engineer team but you’ve just closed a funding round and are planning to quadruple that team in the next eighteen months, hiring a permanent CISO is the right choice — although you may still want someone to get things moving and bridge the gap during the executive search process. On the other hand, a stable company with over a hundred engineers that’s more of a “lifestyle business”, i.e. one that’s not trying to grow exponentially, may find long-term outsourcing still makes sense. The same can apply at a large firm that does development or technical work for other companies and has a small and stable internal technical footprint. Fintech, medical, or defense companies may want to bring governance roles in house sooner, and may need a CISO in house before they can even bring a product to market. They still face the challenge of being able to hire someone with the right background into a small team, and they may find that even if they have an in-house role that’s in the CISO position on paper, long-term external guidance is crucial to growing that person into the role they’ll need to be in as the company grows.

Incident Response

We’ll close with a note on incident response, also known as the unscheduled security work that we excluded from our list at the start of this section. Like detection and penetration testing, incident response is best outsourced at small and medium companies. Even with a large engineering team and a ten or twenty person security team where you’ve brought all your security detection back work in house, you’re unlikely to have significant digital forensics capacity. You can respond to small incidents, but if the entire company is offline with ransomware, you need someone to call. It pays to know who that will be and to have an agreement signed beforehand — this is the difference between an eight hour response time and a $500/hour rate and a 72 hour response time and a $2,000/hour rate. Many companies that offer penetration testing service will also sign a master services agreement where, for a minimum annual spend on testing services, you get a fixed rate and response time for incidents with responders who (in theory) already know your systems. You insurance provider may also bundle incident response with your coverage, and may require you to use one of their covered vendors if they're going to cover incident response work. Make sure you’re comfortable with their vendor, rates, and process before your next insurance renewal.¹¹

Conclusion

Across this and the last essay, we’ve gone through all of the areas where we’ve seen major blockers for teams getting security work done. We’re happy to answer questions from teams facing challenges around this sort of thing — you can book an hour of our time to chat here. Our next pieces will cover security metrics, early choices you can make to make your life easier around security, things to look at in an initial review, and the secure use of coding agents. There’s also a series of essays on different security design patterns in the works. If you’ve got other ideas of things you’d like to see here, let us know.

Systems Structure Limited is a boutique security consultancy, providing fractional CISO services to startups. Over the past nine years, we’ve built security programs for fifteen different companies, and our work has been responsible for hundreds of millions of dollars in capital raised by our clients. If you’re reading this essay because you’re getting ready to start a security program at your startup, let’s talk! If you're a startup considering starting a security program, you can book an hour of our time to talk through issues like these here.

Footnotes

There are choices you can make early on to make your life easier later, but that’s another essay.
Entirely out of scope for this essay, but AI is as often going to be harmful as helpful here — you cannot secure a system you do not understand.
More, likely much more for people in less-legible but often critical glue roles.
The incident response side, especially for large incidents, is less scheduled; we’ll talk about this later.
Yes, and no I don't want to talk about it.
But not, for some unclear reason, application security reviews.
Code written in memory-unsafe languages, cryptographic implementations, and anything safety critical, at least, should all get dedicated security review, either in-house or external.
Given the many hundred-fold increase in computational complexity, it’s reasonable to suggest that these tools are underperforming.
Both is also sadly an option.
In my biased opinion, obviously — but I’ve seen the inside of a lot of small engineering teams and seen all of the failure modes of in-house solo security teams repeatedly, both while coming in as a fractional CISO and before in other roles.
Incident response coverage is in most cases the primary reason it makes sense for startups to purchase cyber insurance.

Eleanor Saitta