All Around The World has worked with a variety of successful companies, many of which are Agile in nature, but when discussion of Agile arises, the conversation often goes like this:
Them: We’re agile.
Us: Ah! Do you use Scrum, XP, something else?
Them (embarassed): Um, er, we just sort of get things done.
And they do! And yet they’re embarrassed by the lack of a name for what they actually do. We often hear something like “we do a modified scrum”, meaning they have sprints and stand ups and that’s about it.
Thus, All Around The World has designed SOC Agile, an Agile methodology that is so agile that it describes what you actually do instead of what you are supposed to do. It’s designed for very rapid delivery, with Just Enough Process to provide order to the chaos. Though we draw inspiration from Scrum, XP, Kanban and Crystal Clear, there’s a good chance that claiming your team is SOC-compliant merely involves reading this and saying “yup, we’re SOC-compliant.”
All bolded terms are explained in the description and also in a short glossary near the end. There aren’t many bolded terms because we focus on process, not terminology. The “SOC Roles” and “SOC Process” sections are all you really need to read to understand SOC Agile. Everything else is just providing the rationale. This documentation of SOC Agile is released under the CC by 2.5 license. This means that you’re free to do anything you want with this documentation, including sell it, so long as you credit the original authors (just linking to this post is fine). See the license description for details.
When teams transition into an Agile environment, the changes can be intimidating. XP practitioners regularly complain about pair programming (and hence don’t do it). Scrum practitioners often ignore retrospectives. System administrators get upset about a “failed sprint” because they had to stop all work to fix a critical security hole.
And yet work still gets done. Developers finish a task and take a new task. Wash, rinse, repeat. We call this “Stream of Consciousness” (SOC) development. By formalizing this we introduce Just Enough Process to get things done in a structured manner. “Just Enough Process” is an appropriate tagline for management. “Shut Up And Write Some Code” is an appropriate tagline for developers.
When to Use SOC Agile
SOC Agile is appropriate for small teams where the entire code base can be roughly understood by a single programmer. That doesn’t mean that someone points to a given library and the developer knows how it’s coded, or exactly what it does, but the developer should at least have a sense of what it’s for. Because SOC is designed to avoid bus-sensitive code, it’s important to have a small to medium size codebase. What this means is that as a developer transitions from Newbie to Veteran (described later), they won’t always know how to solve a problem, but they’ll know where to look to solve that problem.
You may think of SOC Agile as a transitional methodology. This means that you may adopt SOC Agile before your team transitions to a more formal method (it’s particularly well-suited as a transition to Scrum), but many teams find that SOC Agile is all they need.
Benefits of SOC Agile
- High morale: developers can focus on getting stuff done.
- Rapid development: developers can focus on getting stuff done.
- Ease of implementation: developers can focus on getting stuff done.
You may notice a theme in this.
That being said, SOC is not without its risks. These will be discussed on an ad hoc basis as we discuss its components. However, we do not recommend SOC Agile when a potential for real harm occurs. For example, we probably would not use SOC Agile to design guidance systems for planes or medical radiology software. We also do not recommend SOC Agile for highly complex projects in which the majority of developers cannot understand how various components fit together.
Just to make it clear what we’re focusing on, here’s the development workflow for one company we know (you can click on the image for a better view of the horror):
Every task is expected to follow that process. If you have to break a task into new tasks, you’re still expected to follow that process for every subtask. Developers in this company regularly ignore this workflow and still get things done, but management gets frustrated because they can’t see the exact step that every task is at.
SOC Agile strives to implement the following workflow:
We realize that things are not this simple in the real world, but we strongly advise our clients to understand the Platonic Ideal of Agile and constantly strive towards it.
Drawing on work from Alistair Cockburn’s Crystal Clear methodology, we assume that the following three requirements are the most important in developing high-quality products.
- Easy and regular team communication
- Rapid delivery
- Expert customer access
Many experienced individuals feel that if these three requirements are met, a positive-feedback loop is created which allows the team to rapidly build and improve upon products. Any obstacles to these three requirements being met are detrimental to the productivity of any SOC-compliant team.
We are also firm believers in the Pareto Rule: 80% of your benefits will stem from 20% of your actions. SOC Agile attempts to implement that 20% of actions that give you majority of benefits of Agile. In fact, we’ve found that for many smaller teams, attempting to implement the other 80% of actions may actually be detrimental as you spend more time fighting process instead of developing software. This is due to a fundamental tension in many Agile methodologies: it’s an attempt to create processes that are designed to minimize process. It’s a delicate balance that is easy to get wrong.
Cockburn’s (pronounced coe-burn, if you’re curious) team requirement actually said “colocation” of the team, requiring team members to be physically near each other. While this is preferred, in today’s world we recognize that many development teams are geographically diverse. Thus, tools such as IRC, Google Hangouts, Skype, and other communication channels should be employed to ensure that someone is always available to help. We’ve seen SOC work in teams that are also temporally diverse (separated by many time zones), but this does introduce challenges that require extra attention to team communication.
There are only three roles defined in SOC Agile.
- Team Lead
- Product Owner
Developers, of course, develop the product. They take tasks, finish them, and then take new tasks.
The Team Lead (often shortened to “Lead”) is also a developer. The Lead is also the expert who knows the entire system and can be relied upon to answer questions and provide excellent feedback. Developers decide how they’re going to implement something, but if there is a problem, the Lead has final say. The Lead should defer to the Developer whenever possible and offer constructive advice when appropriate. Team Leads support, not direct.
The Product Owner (PO), similar to Scrum, maintains the product backlog. However, there is no “sprint backlog” because there are no sprints. The PO is responsible for knowing the full backlog and being able to answer any business question related to them. It’s preferred if the PO can also answer technical questions as well, but we realize that it’s often hard to find someone who is both a business expert and a technical expert. Thus, the PO focuses on business excellence and the Team Lead focuses on technical excellence.
Additionally, the PO serves as the proxy for the customer. They are the “expert customer” that you should be able to rely on at all times to answer questions about the product. The PO is encouraged to practice an embedded anthropology approach to knowing their customers. This means that they should use the product, but also work directly with real customers (both new and existing) to understand customer needs and desires.
As an aside, the customer should always be referred to as “the customer” and never as “the user.” In English, the word “user” can have a negative connotation and in our experience it is often used in a condescending manner. Customers are people too.
The SOC Process
SOC, as previously mentioned, stands for “Stream of Consciousness.” It’s actually a “Stream of Delivery”, but SOD Agile, while tempting, is probably not the best choice of acronym.
Every day, a 15-minute time-boxed stand up is held. It’s recommended to be at the end of the day to ensure that what you did is fresh in your mind, but this is not a requirement. The stand up begins with the Summary. Before the stand up, the product owner should briefly discuss where you’ve been and where you are going. Any interesting future issues for the team should be mentioned at this time. The benefit of the Summary is that it helps remind developers of the context of their work and better understand upcoming priorities. This makes it easier to choose appropriate tasks.
After the Summary, each team member should explain:
- What they did today
- What they’ll do tomorrow
- Any blockers they have
On a daily basis, every team member is allowed, even encouraged, to improve the code base. Add a Makefile target to rebuild the project. Factor out common code. Move test data into fixtures. The developer is expected to mention this in the stand up and it is the responsibility of the Team Lead and the PO to rein in developers who spend too much time “improving” instead of “developing” (and vice versa!).
Tasks are “things a developer can do.” Every task is done in a separate branch (when appropriate) and when it is done and the tests pass, it is merged back into the “main”, “trunk”, or “integration” branch, or whatever you call it. Tasks are expected to include tests, when feasible, and if merging the code breaks any test, the developer responsible for merging the code should be the person responsible for fixing the tests (again, when feasible). Details about how to manage source control are left to the team.
Tasks should be as short as possible. Any task expected to take over a certain amount of time (a week is a good limit) should, if possible, be broken down into smaller tasks, each of which can add value even if the overall task is cancelled.
When a task is finished, the developer independently chooses their next task, keeping in mind the context provided by the last Summary. They should prioritize tasks assigned to them, high value tasks, and quick tasks. It is the responsibility of the Team Lead and PO to guide developers to suitable tasks if the team member is regularly making poor choices on their next task.
If enough open tasks are open, it is the optional responsibility of the PO to designate Priority tasks. If a new task is to be chosen, they should be chosen from Priority tasks. If a team member is unconvinced of their ability to accomplish any Priority tasks, they should speak to the Team Lead for guidance.
Obviously, if a critical issue arises, any task can be stopped and the Team Lead can assign someone to tackle the issue. There is no formal structure to this because SOC is designed to be as lightweight as possible.
New team members (Newbies) are initially encouraged to focus on a single area to become very comfortable with the business rules. After the Team Lead and PO agree that the developer has become comfortable in their area, they become Veterans. Veterans are strongly encouraged to regularly choose tasks in areas of the code base they have less experience with. If Veterans do not regularly work on different areas of the code, they may find themselves focusing on a small subset of the code base. Not only does this limit the Veteran’s ability to contribute to the code base, it encourages other developers to not work on areas of the code that it’s assumed the Veteran will work on. It is the responsibility of the Team Lead and the PO to ensure that Veterans are regularly choosing tasks outside of their area of expertise in order to spread knowledge of the system.
Team Leads and POs do not promote Newbies to Veterans until the Newbie:
- Appears to understand the business (not the code base!)
- Is writing solid code
- Is writing appropriate tests
Releases are not described in the SOC process, but it’s suitable for both continuous deployment and regular releases (as one might expect from a sprint or iteration).
Note that in our experience, Newbies get promoted to Veterans fairly quickly and in established teams, you generally only have Veterans, thus reducing the distinction between the two.
The idea behind SOC is not to formalize a new methodology, but to put a name on a practice that is already moderately well-known but followed in an almost embarrassed manner. It’s a way of following a set of reasonable development guidelines that can eventually move on to a more appropriate Agile development process. Think of it as “Agile light”, if you will. By putting a name to it, you can say “we are SOC-compliant” instead of “eh, we just get things done.”
In short, SOC eliminates the silo problem of cowboy coding, but allows the extremely rapid development of products. It encourages creativity while having Just Enough Process to keep development on target.
Again, it bears repeating that SOC is not recommended for very large-scale environments. This is not necessarily because it is not suitable for large-scale environments (we know of at least one large company whose successful development process is very similar to this), but given that most of our experience with SOC-like development models has been with smaller code bases, we are hesitant to suggest it for anything else.
By separating developers into Newbies and Veterans, we create the psychological incentive to explore more of the code base. Ambitious developers quickly want to shed the Newbie label and fully participate in the rest of the team’s work. However, by starting out by becoming an expert in one part of the system, the Newbie can quickly gain confidence in their ability to participate and make a meaningful impact on the system.
Hiring for a SOC process is important. You want developers who are comfortable in a fluid environment, but can also appreciate business concerns. There is a tremendous amount of freedom in SOC and it’s very easy for someone to get carried away and try to “perfect” (*cough*) everything at the expense of adding value to the company. Thus, both the Team Lead and the PO are required to keep things in check.
Also, SOC benefits from skilled, conscientious developers. It’s very easy to spend too much time “perfecting” your code, but it’s also easy to spend to much time pushing out new features and ignoring technical debt. This is a hard balance to maintain. SOC does not address this directly because every project is different and every company is different. However, as a guideline SOC suggests that the PO advocate for features and the Team Lead advocate for technical debt management, but each should be able to understand and even agree with the other’s point of view. If you can’t balance these two needs, you’re likely to have long-term problems.
In practice, we find this development model to be very successful for small projects. The pace of development is very rapid and we don’t find that code quality is significantly worse than for other methodologies. The most significant obstacle that we find is appropriately managing the backlog. Typically backlog items are merely added to a ticketing system, such as Jira, RT, Redmine or Trac. This does not scale well unless the PO is competent at managing the tickets and creating appropriate Priority tasks.
Extending SOC Agile
Do you like retrospectives? Do them. Do you like pair programming? That’s fine. Do you want sprints? Not a problem. SOC Agile is designed to describe the core of what successful development teams are already doing. Items outside of this core are deliberately unspecified to allow you to customize SOC Agile to your heart’s content. A guiding principle of SOC Agile should be “whatever works for you.”
You are encouraged, nay, required to customize SOC to meet the needs of your team. See our follow-up for more explanation.
Many teams are already SOC-compliant without even knowing it. If you follow Scrum, you’re probably SOC-compliant. If you follow XP, you’re probably SOC-compliant. If you do ad hoc development, you may already be SOC-compliant. Don’t stress too much about it. We don’t envision SOC-compliance courses. We don’t envision SOC-certifications (though if you pay us enough money, we’re happy to give a one-hour presentation and print up SOC-certifications for you).
- Backlog — Tasks which have not yet been done
- Critical Issue — Something which must be done now
- Just Enough Process – Only enough process to provide structure
- Newbie — A new developer becoming an expert on part of the system
- Priority — Tasks which should be taken before other tasks
- Product Owner (PO) — The proxy for the customer. Maintains backlog and assigns Priorities
- SOC — Stream Of Consciousness development
- Stand up — An end-of-day “what I did, will do, and blockers” explanation for every developer
- Summary — The PO’s business-focused summary of the project status, given immediately prior to the stand up
- Task — Something a developer can do
- Team Lead — The code base expert. Should also understand business concerns
- Team — All developers, including the Team Lead
- Veteran — A developer who is allowed to work on all parts of the system
There is very little terminology and most of this fits what you already know.
Shut up and write some code.
Seriously, there is nothing radical about the above and it mirrors what people are actually doing rather than forcing them to adopt a new methodology all at once. You might view it as a gentle transition to more formal Agile methodologies. Or, if it’s working fine for you, it might remain your Agile methodology. Its very loose structure presents some risks, but a key component of managing risk is being aware of it. As usual, comments on this are very welcome, particularly suggestions for improvement.