The cybersecurity landscape is a relentless battlefield. Attackers are constantly innovating, churning out new threats at an alarming rate. Traditional security solutions are struggling to keep pace. But fear not, weary defenders! Artificial Intelligence (AI) and Machine Learning (ML) are emerging as powerful weapons in our arsenal, offering the potential to revolutionize cybersecurity.
The Numbers Don’t Lie: Why AI/ML Matters
Security Incidents on the Rise: According to the IBM Security X-Force Threat Intelligence Index 2023 https://www.ibm.com/reports/threat-intelligence, the average organization experienced 270 data breaches in 2022, a staggering 13% increase from the previous year.
Alert Fatigue is Real: Security analysts are bombarded with a constant stream of alerts, often leading to “alert fatigue” and missed critical threats. A study by the Ponemon Institute found that it takes an average of 280 days to identify and contain a security breach https://www.ponemon.org/.
AI/ML to the Rescue: Current Applications
AI and ML are already making a significant impact on cybersecurity:
Reverse Engineering Malware with Speed: AI can disassemble and analyze malicious code at lightning speed, uncovering its functionalities and vulnerabilities much faster than traditional methods. This allows defenders to understand attacker tactics and develop effective countermeasures before widespread damage occurs.
Prioritizing the Vulnerability Avalanche: Legacy vulnerability scanners often generate overwhelming lists of potential weaknesses. AI can prioritize these vulnerabilities based on exploitability and potential impact, allowing security teams to focus their efforts on the most critical issues first. A study by McAfee found that organizations can reduce the time to patch critical vulnerabilities by up to 70% using AI https://www.mcafee.com/blogs/internet-security/the-what-why-and-how-of-ai-and-threat-detection/.
Security SIEMs Get Smarter: Security Information and Event Management (SIEM) systems ingest vast amounts of security data. AI can analyze this data in real-time, correlating events and identifying potential threats with an accuracy far exceeding human capabilities. This significantly improves threat detection accuracy and reduces the time attackers have to operate undetected within a network.
The Future of AI/ML in Cybersecurity: A Glimpse Beyond
As AI and ML technologies mature, we can expect even more transformative applications:
Context is King: AI can be trained to understand the context of security events, considering user behaviour, network activity, and system configurations. This will enable highly sophisticated threat detection and prevention capabilities, automatically adapting to new situations and attacker tactics.
Automating Security Tasks: Imagine a future where AI automates not just vulnerability scanning, but also incident response, patch management, and even threat hunting. This would free up security teams to focus on more strategic initiatives and significantly improve overall security posture.
Challenges and Considerations: No Silver Bullet
While AI/ML offers immense potential, it’s important to acknowledge the challenges:
Explainability and Transparency: AI models can sometimes make decisions that are difficult for humans to understand. This lack of explainability can make it challenging to trust and audit AI-powered security systems. Security teams need to ensure they understand how AI systems reach conclusions and that these conclusions are aligned with overall security goals.
Data Quality and Bias: The effectiveness of AI/ML models heavily relies on the quality of the data they are trained on. Biased data can lead to biased models that might miss certain threats or flag legitimate activity as malicious. Security teams need to ensure their training data is diverse and unbiased to avoid perpetuating security blind spots.
The Takeaway: Embrace the Future
Security practitioners and engineers are at the forefront of adopting and shaping AI/ML solutions. By understanding the current applications, future potential, and the associated challenges, you can ensure that AI becomes a powerful ally in your cybersecurity arsenal. Embrace AI/ML, and together we can build a more secure future!
In a move announced on March 20th, 2024, Redis, the ubiquitous in-memory data store, sent shockwaves through the tech world with a significant shift in its licensing model. Previously boasting a permissive BSD license, Redis transitioned to a dual-license approach, combining the Redis Source Available License (RSAL) and the Server Side Public License (SSPL). This move, while strategic for Redis Labs, has created ripples of concern in the SAAS ecosystem and the open-source community at large.
The Split: From Open to Source-Available
At its core, the change restricts how users, particularly cloud providers offering managed Redis services, can leverage the software commercially. The SSPL, outlined in the March 24th press release, stipulates that any derivative work offering the “same functionality as Redis” as a service must also be open-sourced. This directly impacts companies like Amazon (ElastiCache) and DigitalOcean, forcing them to potentially alter their service models or acquire commercial licenses from Redis Labs.
A History of Licensing Shifts
This isn’t the first time Redis Labs has ruffled feathers with licensing changes. As a 2019 TechCrunch article [1] highlights, Redis Labs has a history of tweaking its open-source license, sparking similar controversies. Back then, the company argued that cloud providers were profiting from Redis without giving back to the open-source community. The new SSPL appears to be an extension of this philosophy, aiming to compel greater contribution from commercial users.
SAAS Providers in a Squeeze
For SAAS providers, the new licensing throws a wrench into established business models. Modifying core functionality to comply with the SSPL might not be feasible, and open-sourcing their entire platform could expose proprietary code. This could lead to increased costs for SAAS companies, potentially impacting end-user pricing.
Open Source Community Divided
The open-source world is also grappling with the implications. While the core Redis functionality remains open-source under RSAL, the philosophical shift towards a more restrictive model has some worried. The Linux Foundation even announced a fork, Valkey, as an alternative, backed by tech giants like Google and Oracle. This fragmentation could create confusion and slow down innovation within the open-source Redis ecosystem.
The Road Ahead: Uncertainty and Innovation
The long-term effects of Redis’s licensing change remain to be seen. It might pave the way for a new model for open-source software sustainability, where companies can balance community development with commercial viability. However, it also raises concerns about control and potential fragmentation within open-source projects.
In conclusion, Redis’s licensing shift presents a complex scenario. While it aims to secure Redis Labs’ financial future, it disrupts the SAAS landscape and creates uncertainty in the open-source world. Only time will tell if this is a necessary evolution or a roadblock to future innovation.
Note: I started this article last weekend to try and explain the attack path “Midnight Blizzard” used and what Azure admins should do to protect themselves from a similar attack. Unfortunately, I couldn't complete/publish it in time and now there is another breach at Microsoft. (🤦🏿) Now, I had to completely redraft it and change the focus to a summary of data breaches at Microsoft and a walkthrough on the current breach. I will publish the Midnight Blizzard defence later this week.
The Timeline of the Breaches
20th-25th September 2023: 60k State Department Emails Stolen in Microsoft Breach
12th-25th January 2024: Microsoft breached by “Nation-State Actors”
11th-14th February 2024: State-backed APTs are weaponising OpenAI models
16th-19th February 2024: Microsoft admits to security issues with Azure and Exchange servers.
Date/Month
Breach Type
Affected Service/Area
Source
February 2024
Zero-day vulnerabilities in Exchange servers
Exchange servers
Microsoft Security Response Center blog
January 2024
Nation State-sponsored attack (Russia)
Email accounts
Microsoft Security Response Center blog
February 2024
State-backed APTs are weaponising OpenAI models
Not directly impacting MS services
July 2023
Chinese Hackers Breach U.S. Agencies Via Microsoft Cloud
Azure
The New York Times, Microsoft Security Response Center blog
October 2022
BlueBleed Data Leak, 0.5 Million user data leaked
User Data
December 2021
Lapsus$ intrusion
Source code (Bing, Cortana)
The Guardian, Reuters
August 2021
Hafnium attacks Exchange servers
Exchange servers
Microsoft Security Response Center blog
March 2021
SolarWinds supply chain attack
Various Microsoft products (indirectly affected)
The New York Times, Reuters
January 2020
Misconfigured customer support database
Customer data (names, email addresses)
ZDNet
This is a high-level summary of breaches and successful hacks that got reported in the public domain and picked up by tier 1 publications. There are at least a dozen more in the period, some are of negligible impact, and others are less probable
Introduction:
Today, The digital landscape is a battlefield, and even tech giants like Microsoft aren’t immune to cyberattacks. Understanding recent breaches/incidents and their root causes, and effective defence strategies is crucial for Infosec/IT and DevSecOp teams navigating this ever-evolving threat landscape. This blog post dives into the security incidents affecting Microsoft, analyzes potential attack paths, and equips you with actionable defence plans to fortify your infrastructure/network.
Selected Breaches:
January 2024: State actors, purported to be affiliated with Russia leveraged password spraying and compromised email accounts, including those of senior leadership. This highlights the vulnerability of weak passwords and the critical need for multi-factor authentication (MFA).
January 2024:Zero-day vulnerabilities in Exchange servers allowed attackers to escalate privileges. This emphasizes the importance of regular patching and prompt updates to address vulnerabilities before they’re exploited.
December 2021: Lapsus$ group gained access to source code due to misconfigured access controls. This underscores the importance of least-privilege access and regularly reviewed security configurations.
Other incidents:Supply chain attacks (SolarWinds, March 2021) and data leaks (customer database, January 2020) demonstrate the diverse threats organizations face.
Attack Paths:
Understanding attacker motivations and methods is key to building effective defences. Here are common attack paths:
Social Engineering: Phishing emails and deceptive tactics trick users into revealing sensitive information or clicking malicious links.
Software Vulnerabilities: Unpatched software with known vulnerabilities offers attackers an easy entry point.
Weak Passwords: Simple passwords are easily cracked, granting access to accounts and systems.
Misconfigured Access Controls: Overly permissive access rules give attackers more power than necessary to escalate privileges and cause damage.
Supply Chain Attacks: Compromising a vendor or partner can grant attackers access to multiple organizations within the supply chain.
Defence Plans:
Building a robust defense requires a multi-layered approach:
Patch Management: Prioritize timely patching of vulnerabilities across all systems and software.
Strong Passwords & MFA: Implement strong password policies and enforce MFA for all accounts.
Access Control Management: Implement least privilege access and regularly review configurations.
Security Awareness Training: Educate employees on phishing, social engineering, and secure password practices.
Threat Detection & Response: Deploy security tools to monitor systems for suspicious activity and respond promptly to incidents.
Incident Response Planning: Develop and test a plan to mitigate damage, contain breaches, and recover quickly.
Penetration Testing: Regularly test your defenses by simulating real-world attacks to identify and fix vulnerabilities before attackers do.
Network Segmentation: Segment your network to limit the potential impact of a breach by restricting access to critical systems.
Data Backups & Disaster Recovery: Regularly back up data and have a plan to restore it in case of an attack or outage.
Stay Informed: Keep up-to-date on the latest security threats and vulnerabilities by subscribing to security advisories and attending industry conferences.
Conclusion:
Cybersecurity is an ongoing battle, but by understanding the tactics employed by attackers and implementing these defence strategies, IT/DevOps admins can significantly reduce the risk of breaches and protect their networks and data. Remember, vigilance and continuous improvement are key to staying ahead of the curve in the ever-evolving cybersecurity landscape.
Disclaimer: This blog post is for informational purposes only and should not be considered professional security advice. Please consult with a qualified security professional for guidance specific to your organization or mail me for an obligation free consultation call.
Nobody plans to leave AWS in the startup world, but as they say, “sh** happens.”
As engineers, when we write software, we’re taught to keep it elegant by never depending directly on external systems. We write wrappers for external resources, we encapsulate data and behaviour and standardise functions with libraries.
But, When it comes to the cloud… “eerie silence”
Companies have died because they needed to move off AWS or GCP but couldn’t do it in a reasonable and cost-effective timeline.
We (at Itilite) had a close call with GCP, which served as our brush with the fire. Google had arguably one of the best Distance Matrix capabilities out there. It was used in one of our core logic and ML models. And on one fine Monday afternoon, I have to set up a meeting with my CEO to communicate that we will have to spend ~250% more on our cloud service bill in about 60 days.
Actually, google increased the pricing by 1400% and gave 60 days to rewrite, migrate, move out or perish!
The closest competitor in terms of capability was DistanceMatrix and a reliable “Large” player was Bing. But, both left a lot for in the “Accuracy”. So, for us, the business decision was simple: make the entire product work in “Reduced Functionality” mode for all or start differential pricing for better accuracy! In either case, those APIs must be rewritten with a new adaptor.
It is not an enigma why we do this. It’s simple: there are no alternatives, there is no time to GTM, But maybe there is. I’ll explain why you should take cloud-agnostic architecture seriously and then show you what I do to keep my projects cloud-agnostic.
Cloud Service Rationalisation
The prime reason you should consider the ability to switch clouds and cloud services is so you can choose to use the cloud service that is price and performance-optimized for your use case.
When I first got into serverless, we wrote a transformative API on Oracle Cloud (Bcoz we were part of their Accelerator Program and had a huge credit.) but it fed part of the data that the customer-facing API relied on.
No prize for guessing what happened?
It was a horrible mistake. Our API had an insane latency problem. Cold start requests added additional latency of at least 2 seconds per request. The AWS team has worked hard to build a service that can do things that GCP’s Cloud Functions simply can’t, specifically around cold starts and latency.
I had to move my infrastructure to a different service and a revised network topology.
Guess we would have learned the problem by now, but as we will find out, we did not.
This time it was a combination of Kafka and the AWS Lambda that created an issue. We had relied on Confluent’s connectors for much of the workload interfaces and had to shell out almost $1000 per month per connector!
Avoiding the Cloud Provider Killswitch
Protect Your Business from Unexpected Termination
As a CXO, you may not be aware that cloud providers like AWS, GCP, and Azure reserve the right to terminate your account and destroy your infrastructure at any time, effectively shutting down your business operations. While this may seem like an extreme measure, it’s important to understand that cloud providers have strict terms of service that can lead to account termination for a variety of reasons, even if you’re not engaged in illegal or harmful activities.
A Chilling Example
I recently spoke with a friend who is the founder of a fintech platform. He shared a chilling incident that highlights the risks of relying on cloud providers. His team was using GCP’s Cloud Run, a container service, to host their API. They had a unique use case that required them to call back to their own API to trigger additional work and keep the service active. Unfortunately, GCP monitors this type of behaviour and flags it as potential crypto-mining activity.
On an ordinary Sunday, their infrastructure vanished, and their account was locked. It took them six days of nonstop effort to migrate to AWS.
Protect Your Business
This incident serves as a stark reminder that any business operating on cloud infrastructure is vulnerable to unexpected termination. While you may not be intentionally engaging in activities that violate cloud provider terms of service, it’s crucial to build your infrastructure with the possibility of termination in mind.
Here are some key steps you can take to protect your business from the cloud provider killswitch:
Read and understand the terms of service for each cloud provider you use.
Choose a cloud provider that aligns with your industry and business model.
Avoid relying on a single cloud provider.
Have a backup plan in place.
Regularly review your cloud usage and ensure compliance with cloud provider terms of service.
By taking these proactive measures, you can significantly reduce the risk of your business being disrupted by cloud provider termination and ensure the continuity of your operations.
Unleash the Power of Free Cloud Credits
For early-stage startups operating on a shoestring budget, free cloud credits can be a lifeline, shielding your runway from the scorching heat of cloud infrastructure costs. Acquiring these credits is a breeze, but the way most startups build their infrastructure – akin to an unbreakable blood oath with their cloud provider – restricts them to the credits granted by that single provider.
Why limit yourself to the generosity of one cloud provider when you could seamlessly switch between them to optimize your resource allocation? Imagine the possibilities:
AWS to GCP: Upon depleting your AWS credits, you could effortlessly migrate your infrastructure to GCP, taking advantage of their generous $200,000 credit offer.
Y Combinator: As a Y Combinator startup, you’re entitled to a staggering $150,000 in AWS credits and a mind-boggling $200,000 on GCP.
AI-Powered Startups: If you’re developing AI solutions, Azure welcomes you with open arms, offering $300,000 in free credits to fuel your AI models on their cloud.
By embracing cloud-agnostic architecture, you unlock the freedom to switch between cloud providers, potentially saving you a significant $200,000 upfront. Why constrain yourself to a single cloud provider when cloud-agnosticism empowers you to navigate the cloud landscape with flexibility and cost-efficiency?
Building Resilience: The Importance of Cloud Redundancy
In the ever-evolving world of technology, no system is immune to failure. Even industry giants like Silicon Valley Bank can outright disappear over a weekend or AWS’ main Datacenter can go offline due to a power fluctuation, highlighting the importance of proactively safeguarding your business operations.
Imagine the potential financial impact of a 12-hour outage on AWS for your company. The costs could be staggering, not only in lost revenue but also in reputational damage and customer dissatisfaction or even potential churn.
This is where cloud redundancy comes into play. By running parallel segments of your platform on multiple cloud providers, such as AWS and GCP, you’re essentially creating a fail-safe mechanism.
In the event of an outage on one cloud platform, the other can seamlessly pick up the slack, ensuring uninterrupted service for your customers and minimizing the impact on your business. Cloud redundancy is not just about disaster preparedness; it’s also about optimizing performance and scalability. By distributing your workload across multiple cloud providers, you can tap into the unique strengths and resources of each platform, maximizing efficiency and responsiveness.
In our case, we run the OCR packages, SAML, and Accounts service on Azure, our core “Recommendation engine” and “Booking Engine” on AWS. Yes, having a multi-cloud will involve initial costs that might be prohibitive, but in the long run, the benefits will far outweigh the costs.
Cloud Cost Negotiation: A Matter of Leverage
In the realm of business negotiations, the ultimate power lies in the ability to walk away. If the other party senses your lack of alternatives, they gain a significant advantage, effectively holding you hostage. Cloud cost negotiations are no exception.
Imagine you’ve built a substantial $10 million infrastructure on AWS, heavily reliant on their proprietary APIs like S3, Cognito, and SQS. In such a scenario, walking away from AWS becomes an unrealistic option. You’re essentially at their mercy, accepting whatever cloud costs they dictate.
While negotiating cloud costs may seem insignificant to a small company, for an organization with $10 million of AWS infrastructure, even a 3% discount translates into substantial savings.
To gain leverage in cloud cost negotiations, you need to establish a credible threat of walking away. This requires careful planning and strategic implementation of cloud-agnostic architecture, enabling you to seamlessly switch between cloud providers without disrupting your operations.
Cloud Agnosticism: Your Negotiating Edge
Cloud-agnostic architecture empowers you to:
Diversify your infrastructure: Run your applications on multiple cloud platforms, reducing reliance on a single provider.
Reduce switching costs: Design your infrastructure to minimize the effort and cost of migrating to a new cloud provider.
Strengthen your negotiating position: Demonstrate to cloud providers that you have alternative options, giving you more bargaining power.
By embracing cloud-agnosticism, you transform from a captive customer to a savvy negotiator, capable of securing favorable cloud cost terms.
Unforeseen Challenges: The Importance of Cloud Agnosticism
In the dynamic world of business, unforeseen challenges (and opportunities) can arise at any moment. We often operate with limited visibility, unable to predict every possible scenario that could impact our success. Here’s an actual scenario that highlights the importance of cloud-agnostic architecture:
Acquisition Deal Goes Through
This happened with One of my previous organisations, we tirelessly built this company from the ground up. Our hard work and dedication paid off when a large SaaS Unicorn approached us with an acquisition proposal.
However, during the due diligence, a critical issue emerged: Our company’s infrastructure was entirely reliant on AWS. The Acquiring company had a multi-year multi-million dollar deal with Azure and the M&A team made it clear that unless our platform can operate on Azure, the deal is off the table!
Our team faced the daunting task of migrating the entire infrastructure to Azure within a limited timeframe and budget. Unfortunately, the complexities of the migration proved time-consuming and the merger took 5 months to complete and the offer was reduced by $2 million!
The Power of Cloud Agnosticism
This story serves as a stark reminder of the risks associated with a single-cloud strategy. Had our company embraced cloud-agnostic architecture, we would have possessed the flexibility to seamlessly switch between cloud providers, potentially leading to a bigger exit for all of us!
Cloud-agnostic architecture offers several benefits:
Reduced Vendor Lock-in: Avoids dependence on a single cloud provider, empowering you to switch to more favourable options based on your needs.
Improved Negotiation Power: Gains leverage in cloud cost negotiations by demonstrating the ability to switch providers.
Increased Resilience: Protects your business from disruptions caused by cloud provider outages or policy changes.
Enhanced Scalability: Enables seamless expansion of your infrastructure across multiple cloud platforms as your business grows.
Embrace Cloud Agnosticism for Business Continuity
In today’s ever-changing technological landscape, cloud-agnostic architecture is not just a benefit; it’s a necessity for businesses seeking long-term success and resilience. By adopting a cloud-agnostic approach, you empower your company to navigate the complexities of the cloud landscape with agility, adaptability, and cost-efficiency, ensuring that unforeseen challenges don’t derail your journey.
My Solution
Here’s what I do about it, now after the lessons learnt. I use Multy. Multy is an open-source tool that simplifies cloud infrastructure management by providing a cloud-agnostic API. This means that developers can define their infrastructure configurations once and deploy them to any cloud provider without having to worry about the specific syntax or nuances of each cloud platform. While Multy provides an abstraction layer for deploying cross-cloud environments, you will also need to incorporate cloud-environment agnostic libraries to really make a difference.
You were scrolling through Twitter or your favourite SubReditt on the latest tech trend and a sudden feeling of FOMO creeps in. You’re not alone.
While the notion of a “10x developer” has traditionally been considered aspirational, the emergence of AI-powered tools is levelling the playing field, empowering developers to achieve remarkable productivity gains. While there might be 1000s of possible “AI tools”, I’ll restrict to tools which could yield a direct productivity boost to a developer’s day-to-day work as well as the outcome.
If you have used GitHub Copilot. Think of a Cody as a Turbocharger for Copilot. If you have not used Copilot, you should first try it. Either of these can understand your entire codebase, code graphs, and documentation and help you write efficient code, write unit tests, and document the codebase for you.
While the claim of a 10x speed increase is not substantiated, it shows clear intent to improve productivity drastically. However, it’s in beta, and the tool acknowledges that it’s not always correct, though they’re making rapid improvements. Yes, GitHub Copilot X is there — but then, your organisation needs to be on the Enterprise plan or you might have to add an additional $10-20 per user per month, and Cody is already here.
2. AI Code reviews – Offload the often mundane task of code reviews
While CodeRabbit and DeepCode (now acquired by Snyk) are some of the trailblazers in this space, I have not had the opportunity to work with either of them for any stretch of time. If you know about their relative strengths or benefits, please add a comment, and I will incorporate it. The tool I use most regularly is called Robin-AI-Reviewer, from the good folks at Integral Healthcare (funded by Haystack). My reasoning is two-fold, It is open-source and if it is good enough for HIPPA-compliant app development and certification assessment, it’s a good starting point.
3. AI Test writing – Delegate the task of writing tests to AI- CodiumAI
CodiumAI serves as an AI test-writing assistant. It analyses your code, docstrings, and comments to suggest tests intelligently. CodiumAI addresses a critical aspect of software development that often consumes valuable time: testing. While numerous tools prioritize code writing and optimization, ensuring code functionality is equally vital. CodiumAI seamlessly fills this gap, and its intelligent test generation capability can substantially enhance development efficiency and maintain superior code quality.
4. AI Documentation Assistant — Get AI to write docs for you
This is a no-brainer, who loves writing code walkthroughs and docs? No? Didn’t think so! Mintlify serves as your team’s technical writer. It reads and interprets your code, turning it into a clear, readable document. By all accounts, it is a definite must.
Disclaimer: I have not personally used this and have been mostly able to get this done with Cody, itself. And then, I am no longer doing the primary documentation as my main responsibility.
5. AI Comment Assistant – Readable AI — Never write comments again
Readable AI automates the process of generating comments for your source code. It’s compatible with several popular IDEs, like VSCode, Visual Studio, IntelliJ, and PyCharm, and it can read most languages.
6. AI Tech Debt Assistant – Grit.io
Grit.io is an automated technical debt management tool. Its prime function is auto-generating pull requests that manage code migrations and dependency upgrades. Grit is in beta and available for free till beta moves to RC1. But it actually has about 50+ pattern libraries and it is growing.
I absolutely love it and Grit alleviates a significant portion of the manual work involved in managing migrations and dependency upgrades. They say it 10x’s the refactoring and migration process. I’d say at 33% of what they say, It will still be 300% of what productivity increases. And it is a considerable gain. If you’re an Engineering Leader and you have a “Budget” for 1 tool only, It should be this!
7. AI Pull Request Assistant – An “AI” powered DIff tool
What The Diff AI is an AI-powered code review tool. It writes pull request descriptions, scrutinises pull requests, identifies potential risks, and more. What The Diff claims to be able to significantly speed up development timelines and improve code quality in the long run. It could take a great deal of pain out of the process.
Disclaimer: I have not personally used this
8. AI-driven residential Wizard – Adrenaline AI — Explain it to me
Adrenaline AI helps you understand your codebase. The tool leverages static analysis, vector search, and advanced language models to clarify how features function and explain anything about it to you. The thing I like about this tool very much is, it can be leveraged to automate the “How tos” for your software engineering teams!
9. AI collaboration companion for software projects
Stepsize AI by Stepsize is an AI companion for software projects. It seamlessly integrates with tools like Slack, Jira, and GitHub, providing insightful overviews of your activities and offering strategic suggestions.
The tool uses a complex AI agent architecture, providing long-term “memory” and a deep understanding of the context of your projects.
10. AI-Driven Dev Metrics Collection – Hivel.ai
While strictly speaking, not an AI-driven “assistant” to an average developer, I feel it is nevertheless a good tool for the Engineering org and Engineering leaders to keep track and make course corrections. It provides a Cockpit/Dashboard of all the metrics that matter.
Hivel is built by an awesome team of devs and led by Sudheer
How to Manage Technical Debt in 2023: A Guide for Leadership
In this article, I will summarise effective strategies and best practices to tackle tech debt head-on.
Technical debt is an inevitable reality in software development. But, it can be leveraged just like a financial loan/debt can help you achieve your goals, if managed properly. It can be used to drive competitive advantage by allowing companies to launch new products and features faster, experiment with new technologies, and improve the scalability and performance of their systems. However, like all loans, it need to be “Repaid” properly and at the right time, failing on it will create a downward spiral.
If you’re not careful, technical debt can quickly become a major burden that slows down development and makes it difficult to add new features or even fix bugs in a timely manner.
We will discuss how to identify technical debt and the signs of poorly managed debt, and then provide a strategy for reducing it. We will also discuss what a healthy level of technical debt looks like and how leaders can use it to their advantage.
Bad debt takes money out of your pocket, while good debt puts money in your pocket.
– Robert Kiyosaki
The same is true of tech debt.
Technical debt is the cost of not doing things the right way the first time. Good technical debt is accrued when you make trade-offs to meet deadlines or deliver new features quickly.Bad technical debt is accrued when you make poor decisions or cut corners.
Bad tech debt will probably make your PMs, Sales and CEO happy for a quarter or two. But after that, they will be asking why everything is behind schedule and dealing with customer complaints because things aren’t working properly.
Now that I have presented the obvious in a familiar “Quadrant”, you can actually skip the terminologies and definitions part of this article! 😀
For my verbal brethren, Which is the Tech Debt you’d need to ruthlessly hunt down to extinction? Obviously, it is the untracked, undocumented ones. And the ones which are dragging your team on a downward spiral (immaterial of whether it is tracked or not)
Why does your Tech Debt keep accumulating?
Before we can think about building a strategy to solve tech debt, we need to understand how it gets out of control in the first place.
It’s called “impact visibility”.
Fixing code debt issues is impossible if:
1, You’ve no record of what technical debt issues you have
2, You’ve got a backlog, but you can’t see which issues are related to what code
In both cases, you can’t prioritise tech debt over shipping new features.
We need to get more granular about what impacts these two tech debt cases above.
Issue invisibility — There’s no source of shared knowledge. Codebase health info is locked in (few) engineers’ heads.
No code quality culture — Shipping fast, whatever the cost, like it’s going out of fashion.
Poor process — Tech debt work sucks. Nobody likes creating Jira tickets. “Jira” has become a dirty word.
Low-time investment — Justifying the time to fix tech debt or to refactor is a constant uphill battle. After a point, engineers become silent!
Lack of context — Issues in Jira are a world away from the hard reality of the codebase. They’re not related in any way.
So what’s the source of this? Let’s talk strategy.
Spoiler… It’s about changing organisational culture and developer behaviour to track issues properly.
Creating a strategy to reduce technical debt
Track. Issues. Properly.
Good tech debt management starts with team-wide excellence at tracking issues.
You can’t have a tech debt strategy without tracking.
The engineering leader’s job is to make that “issue tracking” easy for your team. There is supposed to be a software for that – Jira, Asana, Rally or something of that sort.
The problem is, I’ve never believed they really get to the bottom of the problem, and after speaking with scores of engineers and leaders about it, they usually don’t either. My personal belief is most companies suffer on the velocity after their Jira rollout! It is a bit like,
No two countries that both have a McDonald’s have ever fought a war against each other.
Show engineers when they’re working on code with tech debt, without them having to jump thru 3 hoops.
Make it really easy for team members to report tech debt.
Create a natural way to discuss codebase issues.
Integrate tech debt work into your workflows and involve PMs if required.
There are multiple ways to achieve this, the easiest is to not address it. Ie: not address it intentionally, just tweak your existing pipeline. This can be done by,
A very robust linting & integration to the IDE
Tighter Git rules for commits
SAST which runs on the pipeline
and can feed into the IDE
Prioritising impactful tech debt
At this point, it should be obvious, but prioritising the right issues is only possible if you’re tracking the impact of these “issues” and it could be direct or indirect (Dependency, Sequencing, Rework avoidance etc) .
Once you’ve got them, you should regularly and consistently use them to decide what to address. This usually happens during the backlog grooming or sprint planning sessions. But, this decision-making process needs to be strategic. Not at all tactical, ie: DO NOT delegate it to the whims and whimsicals of your TL/PM or even EM.
You or someone with a context of the organisation and position on sales, clients, revenue etc., should be doing this.
A good way to start is by choosing a theme each time you prioritise issues. For example, you could prioritise issues that…
Are impacting a specific feature you need to work on in the next quarter
Are impacting the customer’s UX
Are affecting efficiency/morale on the team
Are impacting the security posture
This is often straightforward if you’ve got high-quality issues that traceable to code and tagged as such.
Most people wonder how to get the time for these “Tasks”. I have two recommendations.
Take an entire sprint every quarter to repay the tech debt (Will need high-level buy-in, It is slightly harder to align your CXOs)
Allocate 15-20% of bandwidth in every sprint. (Easier to achieve buy-in from CXOs, harder to drive with engineers)
Engineers generally won’t prioritise tech debt work by themselves because of the conflict of interest/pressure of shipping fast. This was evident from multiple high velocity/impact software engineering teams including ones at AirBnb, Netflix and Spotify. A commitment to code refactoring and maintenance work should be endorsed and supported from the top and reinforced regularly.
How much Tech Debt can you take on?
Managing technical debt is like managing financial debt. You can use it to your advantage, but you need to be careful not to let it get out of control.
Your technical debt budget is the amount of technical debt that you are willing to take on in order to achieve your business goals. You should not try to solve all of your technical debt at once, but instead focus on the most important items.
Prudent technical debt is debt that you take on deliberately and knowingly, in order to achieve a specific goal. For example, you might take on technical debt to launch a new product quickly, or to add a new feature that is in high demand by your customers.
If you manage your technical debt properly, it can be a powerful tool for gaining a competitive advantage. However, if you let your technical debt get out of control, it can lead to serious problems, such as increased costs, delays, and security vulnerabilities.
Concluding remarks:
Technical debt is one of the most neglected areas of software development. It is often only given priority when it is too late and has already caused serious problems.
However, when leaders work together and develop a consistent and process-driven strategy, technical debt can be effectively managed.
The best engineering teams are constantly thinking about how to use their technical debt budget to their advantage.
Disclaimer: I have been an enormous proponent of Developer Productivity and have tried to implement automated metrics collection in 3 orgs with varied success. In my Mentoring sessions with early-stage startup leaders as well, I (re)enforce the importance of being aware of Dev Productivity. So much so, that I have written a 2-part article on the same here, here and here. I have also been a huge fan of McKinsey and how they seem to get answers which eluded the attention and resources of mega-corporations or governments alike. However, this article is written to communicate an entirely different perspective. In my opinion, McKinsey has got this entire “framework” thing about “dev productivity” wrong.
Introduction:
About a month back, McKinsey published an article claiming that they have developed a framework to measure productivity. They also acknowledged the fact that they were simply rehashing some of the existing metrics (like DORA and SPACE), which were used by Engineering Leaders and have simplified it (without the context) and are pitching it to their traditional buyers, the C-Suite executives in Mega corporations. Actually, some of these metrics can be useful tools if used correctly -One example is Hand-offs. But, the main reason I have chosen to write this article is their central focus seems to be “Coders should code”. It also appears to have A) missed the context of every metric, OR B) Omitted the context so as not to burden their target audience.
Finally, there is a mix-max of things to track, metrics to monitor and Opportunities to Focus, which looks like
Captain Ramius Pointing to a young Jack Ryan that Admiral Halsey was Stupid!
The Legendary Kent Beck has written a deep 2-part piece on countering the conjectures presented by McKinsey and elaborating on the gaps that engineering orgs are traditionally bound to manifest. It is very well written and covers almost everything. There are also a bunch of other eminent Software Engineers who have written on this and I have tried to give a quick lot at the bottom of this article.
What Was I concerned about?
Focus On Activities
I was primarily concerned about the lack of focus on Outcomes and Impact and a focus on the “Activities” in the proposed framework!
Any engineering leader or manager will tell you that Code Review Velocity and Deployment Frequency have nothing to do with measuring outcomes. While I will not discount Cycle Time or MTTR (I take pride in building multiple teams with one of the lowest MTTR and Cycle times in the ecosystem). They are indicators of some process elements/activities that could lead to outcomes. If we want to measure something, it should be Outcomes, not activities!
Focus on Optimisation of Irrelevant Metrics
Code Review Velocity:
If you want to time-motion the code review process in the entire stream map, you’ll find that async code review is killing your productivity. Pairing improves that dramatically. Instead of trying to sub-optimize for code review, measure the thing we actually want to improve. Which will be “Cycel Time”.
Story Points Completed:
Let’s agree on a basic fact. A “story point” is a made-up number. It was conceived as yet another way to obfuscate estimates for thought work that is difficult to estimate. As originally conceived, it represented the number of mythical “ideal days” of effort. There’s so much time wasted on getting better at “story pointing,” arguing about the Fibonacci sequence, “planning poker,” and other story point nonsense. Frankly, it is one of the “Bad” elements of Scrum! As a leader, you should find and remove handoffs and wait times. Story points are useless for anything and even more useless for this goal. Track throughput instead.
Handoffs:
This is a good one. Good job, McKinsey. You got something right. Stop using testing teams, use pairing instead of code review, operate what you build, and don’t have any people doing anything manual to the right of development.
Contribution Analysis and Opportunities focus
In the other focus areas, they have listed metrics at the individual level that can be useful unless you measure “developer satisfaction,” “retention,” and “interruptions” at the individual level. These should only be measured in aggregates to prevent any cognitive bias. IMO, Things start getting really toxic in the “Opportunities focus” section, though.
I have been part of organisations and processes where there was a focus on tracking and measuring the outcomes of individuals. It did not play out well, ever. My Conclusion after reading the article for the second time is that McKinsey thinks their intended audience (CEOs and CFOs) cannot understand “systems thinking.” Now, If you roll out this or a similar framework and announce this and what do you think will happen?
You have a group of people all working on the same backlog but not acting as a team. Code review suffers, mentoring sufferers, pairing is hard, work breakdown suffers, etc. Anything that requires more than one person to conduct/conclude, including helping someone get unstuck, will get deprioritised!
Overall, The inferences seem to be based on hard facts, but the conjectures are all flawed.
Why This Now?
At this point, I want to highlight what “Triggered” me to write this, read the following.,
For example, one company found that its most talented developers were spending excessive time on noncoding activities such as design sessions or managing interdependencies across teams. In response, the company changed its operating model and clarified roles and responsibilities to enable those highest-value developers to do what they do best: code.
McKinsey’s Article on the purported Framework
Wow. I pray for that company.
So, I believe after McKinsey pointed to the fact, that developers are involved in irrelevant things like design, architecture etc. They created separate towers of responsibility for design. In that case, I am puzzled about who will be responsible for the minor things like dependency management, prerequisites, versioning, capacity planning, concurrency, scalability etc.
Did they get anything Right?
Yes. There are tonnes, but they are buried at the bottom. Their focus on Hand-offs and cycle times are really worth tracking in any engineering org. To the authors’ credit, they have also identified some of the core issues with measuring Developer Productivity. But, someone higher in the firm seem to have suggested to soften the blow. So, they have diluted and buried those sections. I will share 2 gems here.
To truly benefit from measuring productivity, leaders and developers alike need to move past the outdated notion that leaders “cannot” understand the intricacies of software engineering, or that engineering is too complex to measure.
The real problem is that in many large organisations, “The Management” doesn’t understand the work they manage. Management can understand the intricacies of software engineering if they become leaders and study the work they manage. In a large behemoth, not all managers are leaders. They want a framework and will enforce it with an iron fist. Now, McKinsey has delivered them a framework!
Learn the basics. All C-suite leaders who are not engineers or who have been in management for a long time will need a primer on the software development process and how it is evolving.
This one Nailed it! The primary reason “Management” finds it difficult to measure the right thing is because they sometimes do not understand the work they want to measure. Leaders who understand do measure the right things. My primary concern with this framework is, in trying to solve this, McKinsey has made the problem worse!
Just google “McKinsey developer productivity” and you’ll find more articles on how this framework is flawed than the original article link!
Anto’s Response to the Article and the purported Framework.
If you’re a software engineer, manager or leader, I am sure you have heard the term ‘10x developer’ used in discussions. It refers to developers who are purportedly 10 times more productive, or capable, than their peers, while it is a hotly contested category. Some refer to it very liberally, others deny that it even exists. In the last 40+ years, the ‘10X developer` has become a ‘Loch Ness` of the tech world, fueled by the hype associated with Silicon Valley.
I’m not about to delude myself into thinking that writing a blog on it to pass my verdict will put these theories to rest, but the question has gained enough traction that it deserves a little articulation.
Do 10x developers really exist, and if so, how would we distinguish them?
Framing the Issue
All of us can acknowledge that the range of skills in most human activities can be extensive. A marathon runner can cover roughly 10 times the distance that an untrained person could, while a professional chef can cook a 5-course meal in 1/5th the time it takes an average person to do a 2-course meal.
Coding, which is a hugely complex field unencumbered by physical limitations, should naturally show differences in the skill that vary by orders of magnitude. Thus, if by 10x developer we simply mean a person whose skill level is in a different league compared to someone else, then clearly they exist.
How could anyone argue otherwise?
Here’s the rub though, in the data-driven and lexically precise world of modern tech, that’s not what 10x developer means. Instead, a 10x developer is supposed to be someone who genuinely outperforms others by 10 times or more on some quantifiable scale. That ‘quantifiable scale’ is where the problems start.
Tom DeMarco and Tim Lister have conducted the “Coding War Games” since 1977. This is a public productivity survey in which teams of software implementors from different organizations compete to complete a series of benchmarks in minimal time with minimal defects. They’ve had over 600 developers participate. Its results are publicly available and is very informative, to say the least. Jeff Lester published a wonderful piece on the origins of the 10X developer here.
The top findings from these are,
1, Get your working environment Right
The overriding observation from this study is that quiet, private, dedicated working space with fewer interruptions led to groups that performed significantly better.
2, Remove the Net Negative Producing Programmer
Some developers are “net negative producing programmers” (NNPP), that is they produce so many defects that removing them from the team increases productivity. This is the opposite of a 10X developer, these people are the ones that make the team productivity go from bad to worse.
The Problem with Measuring ‘Skill’
Even where skill can vary wildly, differences will not necessarily be quantifiable. A talented artist may know how to create a painting that teleports you to a whole other world compared to an average artiste who can transport you to the scene.
But the question is, can you attach numbers to that painting’s beauty?
The work of a developer isn’t nearly as abstract, but not all of it can be reduced to metrics either, and definitely not the programming skill itself.
A less glamorous approach may be to judge a 10x dev not in terms of skill but in terms of productivity. Someone who can write 500 lines of code when it takes others to write 50 would then fall in that category.
If you know anything about programming, however, you’ve probably already spotted the problem with this line of thinking. Longer code isn’t necessarily more efficient, and for most people there tends to be a positive correlation between how quickly one works and how many bugs one creates.
This is not to say that programmers can’t produce bug-free code much faster than their peers. Where this statement proves fallacious though, is in trying to peg that difference to a single metric. There are myriad factors at play that will affect a developer’s productivity outside of their skill, including their team and the environment they find themselves. In fact, depending on the situation, the 10x tag may be inaccurate because a developer could be programming well over 10 times as much as another and still produce 1/10th of the ‘Outcomes’!
What should be our conclusion? There can be no doubt that the field of programming has its own Mozarts and Vincent Van Goghs, and few would object if these people were described as being ‘orders of magnitude’ better than the rest. But it is important to recognize that this is only afigure of speech, and not something meant to be used according to its precise quantitative meaning.
I can’t presume to speak for the tech industry as a whole, but I for one have noticed a worrying tendency to read the expression ‘10x developer’ literally.
Ultimately this does more harm than good, as it spreads the myth that there is some universal metric whereby every programmer’s value can always be quantified.
Important qualities like creativity, client focus and teamwork are entirely omitted in this way of thinking, which is why my final suggestion is to stop worrying about lofty 10x developers and whether you are, aren’t, or may or not become one.
Simply focus on being the best developer you can be. That will always be enough.
Some say that NoOps is the end of DevOps. Is that really true? If you need to answer this question, you must first understand NoOps better.
Things are moving at warp speed in the field of software development. You can subscribe to almost anything “as a service” be it storage, network, computing, or security. Cloud providers are also increasingly investing in their automation ecosystem. This leads us to NoOps, where you wouldn’t require an operations team to manage the lifecycle of your apps, because everything would be automated.
Picture Courtesy: GitHub Blog
You can use automation templates to provision your app components and automate component management, including provisioning, orchestration, deployments, maintenance, upgradation, patching and anything in between meaning significantly less overhead for you and minimal to no human interference. Does this sound wonderful?
But is this a wise choice, and what are some advantages and challenges to implementing it?
Find out the answers to these questions, including whether NoOps is DevOps’s end in this article.
NoOps — Is It a Wise Choice?
You already know that DevOps aims to make app deployments faster and smoother, focusing on continuous improvement. NoOps — no operations — a term coined by Mike Gualtieri at Forrester, has the same goal at its core but without operations professionals!
In an ideal NoOps scenario, a developer never has to collaborate with a member of the operations team. Instead, NoOps uses serverless and PaaS to get the resources they need when they need them. This means that you can use a set of services and tools to securely deploy the required cloud components (including the infrastructure and code). Additionally, NoOps leverages a CI/CD pipeline for deployment. What is more, Ops teams are incredibly effective with data-related tasks, seeing data collection, analysis, and storage as a crucial part of their functions. However, keep in mind that you can automate most of your data collection tasks, but you can’t always get the same level of insights from automating this analysis.
Essentially, NoOps can act as a self-service model where a cloud provider becomes your ops department, automating the underlying infrastructure layer and removing the need for a team to manage it.
Many argue that a completely automated IT environment requiring zero human involvement — true NoOps — is unwise, or even impossible.
Maybe people are afraid of Skynet becoming self-aware!
NoOps vs. DevOps — Pros and Cons
DevOps emphasizes the collaboration between developers and the operations team, while NoOps emphasizes complete automation. Yet, they both try to achieve the same thing — accelerated GTM and a better software deployment process. However, there are both advantages and challenges when considering a DevOps vs. a true NoOps approach.
Pros
More automation, less maintenance
By automating everything using code, NoOps aims to eliminate the additional effort required to support your code’s ecosystem. This means that there will be no need for manual intervention, and every component will be more maintainable in the long run because it’ll be deployed as part of the code. But does this affect DevOps jobs?
Uses the full power of the cloud
There are a lot of new technologies that support extreme automation, including Container as a Service (CaaS) or Function as a Service (FaaS) as opposed to just Serverless, so most big cloud service providers can help you kickstart NoOps adoption. This is excellent news because Ops can ramp up cloud resources as much as necessary, leading to higher capacity, performance & availability planning compared to DevOps (where Dev and Ops work together to decide where the app can run).
Rapid Deployment Cycles
NoOps focuses on business outcomes by shifting focus to priority tasks that deliver value to customers and eliminating the dependency on the operations team, further reducing time-to-market.
Cons
You still need Ops!
In theory, not relying on an operations team to take care of your underlying infrastructure can sound like a dream. Practically, you may need them to monitor outcomes or take care of exceptions. Expecting developers to handle these responsibilities exclusively would take their focus away from delivering business outcomes and wouldn’t be advantageous considering NoOps benefits.
It also wouldn’t be in your best interest to rely solely on developers, as their skill sets don’t necessarily include addressing operational issues. Plus, you don’t want to further overwhelm devs with even more tasks.
Security, Compliance, Privacy
You could abide by security best practices and align them with automatic deployments all you want, but that won’t completely eliminate the need for you to take delicate care of security. Attack methods evolve and change each day, therefore, so should your cloud security controls.
For example, you could introduce the wrong rules for your AI or automate flawed processes, inviting errors in your automation or creating flawed scripts for hundreds or thousands of infrastructure components or servers. If you completely remove your Ops team, you may want to consider investing additional funds into a security team to ensure you’re instilling the best security and compliance methods for your environments.
Consider your environment
Considering NoOps uses serverless and PaaS to get resources, this could become a limiting factor for you, especially during a refactor or transformation. Automation is still possible with legacy infrastructures and hybrid deployments, but you can’t entirely eliminate human intervention in these cases. So remember that not all environments can transition to NoOps, therefore, you must carefully evaluate the pros and cons of switching.
So Is NoOps Really the End of DevOps?
TL:DR: NO!
Detail: NoOps is not a Panacea. It is limited to apps that fit into existing #serverless and #PaaS solutions. As someone who builds B2B SaaS applications for a living, I know that most enterprises still run on monolithic legacy apps and even some of the new-gen Unicorns are in the middle of Refactoring/Migration which will require total rewrites or massive updates to work in a PaaS environment, you’d still need someone to take care of operations even if there’s a single legacy system left behind.
In this sense, NoOps is still a way away from handling long-running apps that run specialized processes or production environments with demanding applications. Conversely, operations occur before production, so, with DevOps, operations work happens before code goes to production. Releases include monitoring, testing, bug fixes, security and policy checks on every commit, etc.
You must have everyone on the team (including key stakeholders) involved from the beginning to enable fast feedback and ensure automated controls and tasks are effective and correct. Continuous learning and improvement (a pillar of DevOps teams) shouldn’t only happen when things go wrong; instead, members must work together and collaboratively to problem-solve and improve systems and processes.
The Upside
Thankfully, NoOps fits within some DevOps ways. It’s focused on learning and improvement, uses new tools, ideas, and techniques developed through continuous and open collaboration, and NoOps solutions remove friction to increase the flow of valuable features through the pipeline. This means that NoOps is a successful extension of DevOps.
In other words, DevOps is forever, and NoOps is just the beginning of the innovations that can take place together with DevOps, so to say that NoOps is the end of DevOps would mean that there isn’t anything new to learn or improve.
Destination: NoOps
There’s quite a lot of groundwork involved for true NoOps — you need to choose between serverless or PaaS, and take configuration, component management, and security controls into consideration to get started. Even then, you may still have some loose ends — like legacy systems — that would take more time to transition (or that you can’t transition at all).
One thing is certain, though, DevOps isn’t going anywhere and automation won’t make Ops obsolete. However, as serverless automation evolves, you may have to consider a new approach for development and operations at some point. Thankfully, you have a lot of help, like automation tools and EaaS, to make your transition easier should you choose to switch.
The fact that you clicked on this article tells me that you are leading/heading a Team, group or an entire Engineering function and most likely a fast-paced startup. Assume the following,
It was a regular weekday, and your CEO/CTO asked the most intriguing question.
Do we measure Engineering Productivity? How do we fare? What can we do to improve it?
Well, if your boss’s name is not Elon Musk or if you do not work for Twitter, you can still be saved. Go on and read through. I know it is a long read.
What is Engineering Productivity?
As with anything you’re trying to improve, it starts with measuring the right data. So, you can actually track the right metrics. This data will form the basis of your analysis and baseline. I strongly recommend you don’t change anything about your current engineering process before you can collect sex weeks’ worth of data about your processes. If you start working on processes, you could end up with a Survivorship Basis.
You should have sufficient historical data to make comparisons. On top of that, most teams work in sprints of two weeks, so six weeks of data allows you to collect data for at least three different sprints. This will give you the allowances for any spikes and eliminate any unusual stress or slack on the execution.
Next, you should make gradual changes to the engineering process to see what improves or impedes the value delivery. It’s ideal to only implement one change at a time, so you can see the effect of each change, with all other things being equal. (it never is :D)
For example, if your engineering squads suffer from significant technical debt, you may want to build an additional stub related to feature completion. Every time an engineer completes a new feature, they must document the new feature. This could mean describing the feature, how is it built, what are the outcomes, how it interacts with other functions and the reasoning behind the design decisions.
By continuously measuring engineering productivity metrics, you can determine if this change has positively impacted the developers’ productivity.
How Is Engineering Productivity Measured?
There are potentially 100s of metrics you can measure for an Engineering Org. Here are four key metrics that will help you to get started with measuring engineering productivity. And I have consciously excluded the Sprint Velocity.
4 Prime Directives of Engineering Metrics
1. The One Metrics to rule them all metrics – Cycle Time
Software development cycle time measures the amount of time from work started to work delivered. It is a metric “borrowed” from lean manufacturing, and it is one of the most important metrics for software development teams. In plain speak, cycle time measures the amount of time from the first commit to production release.
2. The Oracle of an Engineering Leader – Release Frequency
You should measure how often you deploy new changes to your customers (production). In addition, you can track deployments to various branches/instances, such as feature branches, hotfix branches, or QA branches. This data would show you how long it takes for a feature/fix to move through the different development stages. In addition, the Release Frequency reflects the throughput of your team. It’s a good stand-in replacement for Agile Velocity, so you don’t spook your Engineers and you are not blind as well.
3. The Guardrail – Number of Bugs
You should definitely track the number of bugs that your team has to resolve within 2 sprints of releasing a feature. This metric helps you to understand the quality of your code better. Higher-quality code should display fewer bugs after feature deployment.
While there are derivative and more evolved metrics like Defect Density, Mean Time to Detect (MTtD), Mean Time to Resolve (MTrR) and Code coverage, those onces makes sense after you’ve taken stock of and address the prime metric “ No: of Bugs” first.
If you want a more detailed list, methodology of QA metrics, refer the links given below.
4. What is your “Blocker” – Review to Merge Time (RTMT)
This may look like a zoom-in on “Cycle time” metric we discussed earlier. But, in fact it is very different. In fact, it is an interesting metric suggested by GitLab’s development handbook.
You should measure the time between asking for a pull request (PR) review and merging the PR. Ideally, you want to reduce the time a feature spends in the review state (or pending review state). A high RTMT prevents developers from progressing while they wait for feedback and encourages context-switching between different issues/features.
Arguably, Context-Switching is the highest productivity killer and should be avoided as much as possible.
So, why would you measure all these engineering productivity metrics?
Why Is Measuring Engineering Productivity Important?
When you’re a “fast-growing startup”, it’s important to keep an eye on engineering productivity. It happens that these startups favour growth through feature delivery at the cost of effectively scaling the engineering team and ensuring the team’s efficiency.
I hear your question.
But, why does my CEO/VP/MD not understand?
Answer is simple
Assume you have to manage multiple VP’s expectations and outcomes (Sales, Marketing, Support etc), Company’s OKRs, and investors (or) board, will you have more time to dedicate to Engineering Productivity?
In these cases, technical debt can quickly grow, which will slowly kill your team’s productivity. Technical debt can have many negative consequences:
More bugs for your team to fix
Lower code quality—not only bugs but also worse code design
Harder to debug code
Scalability issues
A decline in overall happiness and job satisfaction
To avoid all of these scenarios, you should measure the engineering team’s efficiency and avoid technical debt buildup. Avoiding these problems before they occur is an excellent Occam’s razor. But addressing them head-on will have a significant impact on your organisation, both materially and culturally.
In addition to preventing your team’s productivity from going down, the engineering productivity approach allows you to experiment with various approaches to try and improve throughput & efficiency.
So, the goal is to improve the engineering process itself. For example, introducing new tools or applying new techniques. Next, you can measure the impact of these changes on your team’s productivity.
In the next part, I will write down on how can measurement improve engineering productivity, Stay Tuned!
