Brewers CAP Theorem
Computer Scientist, Eric Brewer, stipulated in the theorem that carries his name that you can have two out of three guarantees in distributed data storage with the guarantees being consistency, availability and partition tolerance. The limitations in Brewer’s “CAP” Theorem are not only key in solving modern big-data challenges, they also are critical in solving big-business issues.
Business as a Data Lake
It may be that 6 years running hundreds of experiments across different data philosophies and platforms have created long-term trauma that makes every problem in life look like a big-data problem but I cannot help but notice some parallel lessons between data lakes and organizations.
Business must partition data into their systems and employees. That data must be available to decision makers and it needs to be agreed what data is up to date and correct (consistency). At that high level, it is easy to apply Brewer’s Theorem to how we handle the transmission and replication of data within an organization.
ACID vs BASE
All of us old-timers in the data world grew up believing in the ACID model which values consistency at all costs. Every member of the cluster that needed the information would have the information before it was available to consumers. ACID tactics include locking tables or rows of data (prohibiting access by others) until the update of information is complete. The benefit of ACID philosophy is there in no opportunity for ambiguity between custodians of the data.
BASE philosophies operate knowing that all members do not have the most accurate information at a given time but will at some point. This sacrifices consistency to provide undisrupted availability of information. This approach is called “eventual consistency.”
ACID Meetings
When business meetings require all members to stop other functions and assemble for purposes of exchanging information, consistency is maintained but the participants are not available to use or share the information in their “partition.” The other challenge of these time based “replication cycles” is the members are not always in a state for replication. They may be deep in complex work, dealing with personal issues or fatigue or unable to do the preparation work to process the meetings data. This not only effects business availability it also limits the potential of effective replication and merging of data that a meeting was scheduled to deliver.
Eventual Consistency in Meetings
Before COVID shutdowns, the WitFoo team had already become adept at out-of-band, remote conversations. Our team covers four time zones across the US and each of us have different passions, focus areas and personalities. We have found great success in moving to eventual consistency in data replication to allow us to maximize business availability while delivering a healthy work/life balance.
Tactics of Eventual Consistency Business
The tools we use to maintain a BASE philosophy in data and workflow are:
- Slack (Chat)
- ZenDesk (ticketing)
- Zoom for Video Recording and Meetings
- Vimeo for video hosting
- WordPress LMS
Chat and Ticketing
We like Slack for commenting on everything from the final season of Game of Thrones to how many rows should be written to a Cassandra partition. It provides an archive of all our conversations that we can search and make available to the new members of our growing team that were not yet onboarded when an important exchanged occurred.
Slack isn’t great at keeping track of tasks so internally we use Zendesk ticketing software to assign tasks to each other when we need help on anything. These mechanisms allow WitFookin to pick up tasks and conversations at an ideal time and place.
Zoom for One, Please
We rarely have internal ACID type meetings. If we have ideas to present, we recommend jumping on a Zoom and presenting the ideas to an empty audience while recording. Then share the recording and members will comment when they have the bandwidth to watch and synthesize information.
Now that we’ve been doing this for a while I do find it preposterous that within seconds of receiving new information in a meeting, you are expected to deliver a genius insight. We have had so many break throughs from multi-day irregular Slack conversations born from a phone or recorded presentation.
Giving people the time and space to properly process new information is critical. It also removes the pressure to find a pause from blowhards like me to interject an idea. Record and chat replacements to in person meetings create better thought and innovation while maximizing availability to work on pressing projects and keep personal goals on track.
Use those Voice Minutes
When I really want to test the viability of an idea or really hear another team member’s idea, I put on my headphones, lace my shoes, and go for a “walk and talk.” Webcams are powerful for reading nonverbal emotes but there is power in going through the ancient traditions of verbal story telling. Homing in on a voice as the only interrupt is still a powerful way to exchange ideas without distraction. So much of what we achieved has been on these phone calls and not in front of whiteboards or other tools. Two sharp minds burning through the massive bank of rollover minutes (is that still a thing??) is a great way to bring world changing ideas to clarity. Just don’t forget to write it up and share it with the team so they can have their own walk-and-talks and Slack comments.
Zoom to Hangout
At least once a month we host a movie watch party over Zoom/Plex where we hangout virtually and chat about the movie & life. On the last Friday of every month, we have a Zoom Happy Hour to talk about things we are passionate about. As an extrovert, I do long for human interaction and teams do need to look at each other and let each other know how much they value one another. Those activities do not require PowerPoint, though.
Compaction and Garbage Collection
The tldr; here is: making people assemble together is probably not the most effective (or pleasurable) way to create and replicate information. Taking some lessons from big-data and letting people consume information at a time and pace that aligns with their strengths will make employees more productive, innovative, and happier.
