This was super good James, thanks! I believe all of us faced a "code yellow" problem but never names it properly or never knew it was a thing and how to treat it - thanks for sharing this!
Inspired by the other comment about "how much AI caused it", my question is the other flip of the coin... How much AI helped with it? would you say that nowadays, with how much AI can do/help, shall we be bolder on our scope or just tight on the deadline/exit?
AI is amazing. For example, if you are diagnosing slow queries, pointing AI at it quite often will find bad smells in that endpoint (e.g. n+1 issues because of how the ORM works). Then you can fix that, then easily scan the whole codebase for the same thing and fix those, and then, even better, have AI code review (we use CodeRabbit) spot that pattern on every code change going forward.
Curious about the end - what would you have done if you didn’t reach it? I mean often there are still small mistakes, or things out of your control (unless you go multi cloud).
When you set a goal of 8 weeks with 0 downtime, you were willing for it to take even a year?
Curios about the balance of setting an achievable but ambitious end goal for the code yellow.
Also curious for code red - how do you get out of it and when? In the Google example, not sure they should have gotten out of it 😅
Hey. In terms of the end, if you can't meet the exit criteria of a code yellow in a reasonable amount of time, it turns into a code red. The entire company needs to pivot towards reaching those exit criteria.
I was willing for it to take as long as it needed in order to get to 8 weeks of zero downtime. I know that's ambitious, but it's meant to be ambitious and challenging so that the nature of the work that gets done is good enough to reach where you need to be.
In terms of Code Reds, those are the kinds of things that you get out of when you meet the exit criteria, except those exit criteria are usually big company-wide things. For example, if you have a Code Red around customer churn, you get out of it when customers stop churning at the rate that you expect. Similarly, when there was a Code Red at Shopify around checkouts being able to be hijacked with inline JavaScript injections, the Code Red was exited when everyone had migrated to the new checkout experience that didn't allow you to do that.
Disclosure, I work on operatex.dev, mentioning it since this fits the theme.
The line about waiting for the dramatic failure being already too late is the real takeaway. Most teams have some version of the slow boil happening right now and are calling it a busy quarter instead of what it actually is.
How much was the code yellow caused by vibe coding or agentic workflows running amock? Today's engineering departments seem to be mostly AI. How can you guarentee any kind of SLA when the systems in operation run on statistical probability and inherent confabulation?
This was super good James, thanks! I believe all of us faced a "code yellow" problem but never names it properly or never knew it was a thing and how to treat it - thanks for sharing this!
Inspired by the other comment about "how much AI caused it", my question is the other flip of the coin... How much AI helped with it? would you say that nowadays, with how much AI can do/help, shall we be bolder on our scope or just tight on the deadline/exit?
AI is amazing. For example, if you are diagnosing slow queries, pointing AI at it quite often will find bad smells in that endpoint (e.g. n+1 issues because of how the ORM works). Then you can fix that, then easily scan the whole codebase for the same thing and fix those, and then, even better, have AI code review (we use CodeRabbit) spot that pattern on every code change going forward.
Very interesting read, thanks James!
Curious about the end - what would you have done if you didn’t reach it? I mean often there are still small mistakes, or things out of your control (unless you go multi cloud).
When you set a goal of 8 weeks with 0 downtime, you were willing for it to take even a year?
Curios about the balance of setting an achievable but ambitious end goal for the code yellow.
Also curious for code red - how do you get out of it and when? In the Google example, not sure they should have gotten out of it 😅
Hey. In terms of the end, if you can't meet the exit criteria of a code yellow in a reasonable amount of time, it turns into a code red. The entire company needs to pivot towards reaching those exit criteria.
I was willing for it to take as long as it needed in order to get to 8 weeks of zero downtime. I know that's ambitious, but it's meant to be ambitious and challenging so that the nature of the work that gets done is good enough to reach where you need to be.
In terms of Code Reds, those are the kinds of things that you get out of when you meet the exit criteria, except those exit criteria are usually big company-wide things. For example, if you have a Code Red around customer churn, you get out of it when customers stop churning at the rate that you expect. Similarly, when there was a Code Red at Shopify around checkouts being able to be hijacked with inline JavaScript injections, the Code Red was exited when everyone had migrated to the new checkout experience that didn't allow you to do that.
Got it, thanks for the detailed answer!
Disclosure, I work on operatex.dev, mentioning it since this fits the theme.
The line about waiting for the dramatic failure being already too late is the real takeaway. Most teams have some version of the slow boil happening right now and are calling it a busy quarter instead of what it actually is.
How much was the code yellow caused by vibe coding or agentic workflows running amock? Today's engineering departments seem to be mostly AI. How can you guarentee any kind of SLA when the systems in operation run on statistical probability and inherent confabulation?
0%. Most of the problematic infrastructure issues and patterns even predated ChatGPT.
Finally, someone writes about Incident Management.
Last time I wrote about it (https://substack.com/@jimmypang/note/p-180490537?utm_source=notes-share-action&r=2i7jb6), there was not so much echos - it is sad that people are always after new shinny things only and overlook what really matters.