As I have had opportunity to demonstrate our product to cybersecurity veterans I am often asked “How did your very small team do this when larger, well-funded teams cannot?” It is true, the WitFoo development team has never been larger than 5 active members at any time and we have only had 10 contributors to the code-base. We don’t Frankenstein together open source code, we custom build it all. All told, our code consists of more than 4 million lines of proprietary code written by a handful of hard-hitting warrior developers. As we wrap our newest and grandest release, I’d like to share some insight into how we pulled it off.
Audacious and Tangible Mission
The primary ingredient in building a Special Forces of developers is having the right mission. I’m not talking about cheesy corporate mission statements or platitudes. I mean something grand that changes the course of human history. Something that is going to require pain, fear, sacrifice and exhaustion; a mission that will produce ridicule, attacks and enemies. Powerhouse warriors need an audacious mission. They need a reason to bleed.
In addition to being audacious, the mission must be tangible. All members of the team must know what victory looks like. This informs all the actions that are taken. At WitFoo our mission is to “Mature the craft of cybersecurity operations by providing tools to stabilize it’s seven broken conversations.” The conversations are:
- Responders cannot understand/process what their tools are communicating.
- Security Managers cannot understand what resources their responders need.
- Security Manager cannot effectively communicate with the broader business.
- Organizations cannot hold vendors accountable.
- Organizations cannot safely share information with each other in an operational way.
- Organizations cannot report cybercrime to Law Enforcement in a way that does not create organization risk.
- Law Enforcement can not effectively prosecute cyber criminals due to insufficient or complex evidence.
Failure Worship
Once you have established what the audacious victory looks like, it is time to build a shrine to Failure. There are limited lessons in studying others’ successes. Everyone should learn best practices and approaches and philosophies. That type of learning is easy and cheap. When you are endeavoring to change the world, however, the lessons you need to discover must be mined from the hard ore of failure. The primary characteristic of a SpecOps developer is a high tolerance for failure and pain. This is also true of those that are driven enough to go through Hell Week at BUDS. You know you are going to get knocked down but unless the failure breaks or kills that warrior, they will always get back up and run back into the fray.
We started WitFoo based on my first patent filing that we thought would allow us to solve 95% of the unstable conversations. About 14 months into research and development, I realized it would barely cover 10% of the problems. The realization hurt so bad that I shed loud, nasty man-tears for the better part of an hour. Then I had a beer and wrote the second innovation which later led to the next patent filing. We have done than 7 times so far. We have also rebuilt the entire platform architecture five full times. Each time it hurt to tears.
The relentless failure, broken bones and exhaustion means you won’t keep contributors for too long. You must plan for a rotation. Good warriors will last a good 6 months in these conditions. There are few people daft enough to sustain beatings for the 3-5 years it will take to build a product that will permanently change human history.
Reality Check
To find failure, you must leave the clean, safe lab and go into the ugly, violent world. Nothing dashes delusions of grandeur faster than cold, hard reality-based data. To get up and learn more, you must find situations that can knock you down and kick your teeth out.
We were very fortunate to have more than two dozen organizations ranging from SMB to the Fortune 500 and Public Sector across more than 15 verticals allow us to get beat up on their networks and by their teams. The cold hard truth is, if we’re not fixing the broken conversation in this organization, we are still short of the mission. We learn from that failure and go back to the drawing board.
Metric-Driven Development
One of the lessons that can be learned from the successes of others is every line of code should have a unit test. All systems should have system tests. Automated testing of every imaginable scenario should exist in DevOps build testing. This is the Basic Training of DevOps that every warrior must complete to achieve competence. DevSpecOps advanced training is “metrics everywhere.”
Every function and method we write generates metrics. Everything including error messages, traces and cycle times. We also gather metrics from the system (CPU, RAM, Disk and Network I/O) and Docker container and JVM metrics. We collect metrics on user workflows to validate they are using the product in anticipated ways. We write these metrics to Kafka then buffer, encrypt, compress and ship them to our data cluster in our data center. Every 15 minutes more than 200 metric checks are run against every deployment. When a metric error is detected (an error, high CPU, etc.) we are notified through Slack. Over the last year, we have collected and analyzed more than 9 billion metrics. These metric failures prompt us to write patches, optimizations as well as new system and unit test.
Summary
Changing the world does not require $100M in venture capital or 30 developers if you can recruit a handful of Special Operations Developers that are pledged to an audacious and tangible mission, willing to give Failure the respect and patience it deserves by finding every morsel of truth that it presents to you in an honest endeavor to accomplish that mission. Good hunting!
