Every once in a while, though, I relent. This happened recently when I presented, to my leadership team colleagues, a short overview of the latest tech trends in AI. The pace of change in AI is hard to keep up with; a quick roundup of recent trends would get this team up to speed, I thought. The focus was on tech trends, not business, economic, or cultural ones driven by AI.

What emerged was a list. The AI tools I consulted (ChatGPT, Gemini, Claude) came up with different lists for the same prompt, so I curated my own top ten. Following the presentation & demo session, a blog post on the theme seemed like a good way to mark the zeitgeist. I asked Claude to expand on each item on the list, and used NotebookLM to fine-tune Claude’s response.

Top ten AI trends: A Jugalbandi between Claude and NotebookLM

1. The Autonomy Era: Agentic AI and the Death of the Interface

The strategic center of gravity has shifted toward “Agentic AI.” Unlike the reactive, prompt-dependent models of 2023, these systems possess the ability to plan, use tools, and execute multi-step workflows with minimal human oversight. We are moving from a world of “Ask and Receive” to one of “Set Goal and Observe.”

The Intern Who Never Sleeps: AI that doesn’t just draft your email but books the flight, files the expense report, and argues with the airline—all while you’re busy having a life.

The core differentiator is the ability to interact directly with APIs and execute plans. This poses an existential threat to traditional SaaS companies that have built their moats around manual user interfaces (UIs). If an agent can bypass the dashboard and perform the task via an API, the “interface” becomes a vestigial organ. Forward-thinking giants like Salesforce and Microsoft are already pivoting, transitioning from platforms you log into to orchestrators that work for you.

The autonomy of these agents is being fueled by a radical change in how we build them, moving from rigid syntax to “Vibe-coding.”

2. The Architect’s Renaissance: Vibe-Coding and High-Level Abstraction

Software development is undergoing its most profound abstraction shift since the invention of the compiler. “Vibe-coding” prioritizes intent over syntax, allowing developers—and increasingly, non-developers—to describe a desired outcome in natural language while the AI manages the implementation.

Grammar is Dead, Vibes are Forever: Software engineering where “I want it to look like a sunset and feel like a fast car” is a valid pull request.

The skill mix is shifting. Navigating APIs is becoming obsolete; the new premium is placed on understanding system architecture and mastering “prompt engineering.” This isn’t just a theoretical shift—GitHub reports that developers using AI tools complete tasks 55% faster, and a 2024 Stack Overflow survey found that 76% of developers are already using or planning to use AI coding tools. The barrier to entry for creating complex software is effectively collapsing.

3. The End of the Hold Music: Human-Grade Voice AI

Voice AI has finally crossed the uncanny valley, achieving near-zero latency and emotional intelligence. With OpenAI’s Advanced Voice Mode hitting 320ms latency, we have reached the threshold of natural human conversation. This is a direct challenge to the $280B Business Process Outsourcing (BPO) industry.

The Customer Service Rep with No Coffee Break: An AI that actually understands your frustration—and has the emotional intelligence to fix it before you can even ask for a manager.

The efficiency gains are staggering. Klarna recently replaced the work of 700 agents with an AI assistant that handled 2.3 million conversations. Not only was customer satisfaction maintained, but resolution times plummeted from 11 minutes to just two.

4. The Unified Interface: MCP and the SaaS Ecosystem Integration

The Model Context Protocol (MCP) is dismantling enterprise data silos and turning LLMs into central operating systems. By allowing models to pull data from Slack, update Jira, and schedule meetings in one conversation, the “SaaS sprawl” is being consolidated into a single interface.

The Great API Peace Treaty: Because your tech stack shouldn’t require a diplomatic envoy just to move a Jira ticket into a Slack channel.

Since its release in late 2024, over 1,000 MCP servers have been created. The strategic reality is clear: users no longer want to hunt through fifteen different SaaS apps; they want one interface that speaks to everything. SaaS companies that fail to integrate with protocols like MCP risk becoming “invisible.”

This complex web of integration is creating a new demand for human expertise in the form of Forward Deployed Engineers.

5. High-Tech, High-Touch: The Rise of the Forward Deployed Engineer (FDE)

There is a profound irony in the AI boom: the more advanced the AI, the more human “boots on the ground” are required to make it work. The “Forward Deployed Engineer” is a hybrid coder-consultant embedded with customers to ensure AI actually reaches production-grade maturity.

The AI Sherpa: Part coder, part consultant, fully embedded in your office until the ‘magic’ actually works in production.

Adopted from the “Palantir Model,” companies like OpenAI and Anthropic are increasingly adopting this high-touch approach. Job postings for these roles grew 340% in 2024, reflecting a reality where enterprise AI implementation cycles still take 6-12 months. The strategic reason? AI tooling isn’t yet mature enough for pure self-service deployment; “out of the box” is still a myth for the enterprise.

The difficulty of implementation is matched only by the difficulty of measurement, leading to new ways of “grading” AI.

6. The Mirror Test: Evaluation Frameworks and ‘LLM as a Judge’

As AI outputs scale, human review becomes a bottleneck. The industry has turned to “LLM as a judge,” where advanced models evaluate the performance of other models across dimensions like helpfulness and honesty.

The AI Grading the AI’s Homework: Because humans are too slow, we’ve hired a robot to tell us if the other robot is hallucinating.

While this introduces a “circularity” risk—where models share the same systemic biases—the efficiency is undeniable. Research shows an 80%+ correlation with human judgment.

7. The Attention Pivot: From SEO to Generative Engine Optimization (GEO)

The fundamental trade of the web—content for traffic—is breaking. Generative Engine Optimization (GEO) is the new battleground as AI “Overviews” provide direct answers, often eliminating the need for a user to ever click a link.

The End of the Click: Why search for a recipe when the AI gives you the ingredients, the steps, and a shopping list without you ever visiting the blog?

Gartner predicts a 25% drop in search volume by 2026. This isn’t a future threat; it’s an active crisis. Publishers are already reporting 20-40% year-over-year traffic declines, with organic click-through rates (CTR) potentially dropping as much as 64%.

While old media models struggle, a new breed of AI-Native companies is thriving on the wreckage.

8. The New Guard: The Explosion of AI-Native Businesses

We are moving beyond “AI-enabled” (adding a chatbot to a website) to “AI-native.” Much like how the smartphone enabled Uber, these companies are built on foundations that were impossible before the current AI era.

Born in the Cloud, Raised by a Model: Companies that don’t just ‘use’ AI, but literally couldn’t exist in a world without it.

With $50B raised by AI-native startups in 2024, the application layer is projected to reach $280B by 2030. This viability is driven by a single, staggering economic reality: the collapsing cost of thought.

9. The Commodity of Thought: Dramatically Lower Costs

The economic democratization of AI is here. Through architectural improvements and fierce competition, the cost of high-end intelligence has dropped by 70-90% in just 18 months.

From Caviar to Flour: High-end intelligence used to be a luxury; now it’s a utility priced in fractions of a penny.

The cost of image generation has seen a dramatic reduction, exemplified by DALL-E 2’s cost of $15.00 per image dropping to $0.02 per image with DALL-E 3. Similarly, LLM token costs have plummeted, with GPT-4’s price of $30 per million tokens now compared to $0.15 per million tokens for GPT-4o mini.

These scale economies enable massive use cases—such as processing millions of daily transactions or images—that were previously economically impossible. This cheap intelligence is now being granted a sensory upgrade.

10. The Sensory Upgrade: Multimodal Everywhere

AI has moved from being a text-based “brain in a jar” to a system with eyes, ears, and a voice. Multimodal models natively understand text, vision, audio, and video simultaneously.

The Computer Finally Has Eyes, Ears, and a Voice: Stop typing descriptions; just show the AI your screen or talk to it while you walk.

The impact on human-computer interaction is immediate: 40% of GPT-4 API calls now involve vision features, and Meta’s multimodal models process over 500 million images daily. We are no longer “typing” into a computer; we are “showing” and “speaking” to our digital environments.

Strategic Imperative for 2026: From Use to Orchestration

These trends converge on a single, sobering reality: the bottleneck to innovation is no longer the intelligence of the model, but the gap between the model and the implementation on the ground.

As the cost of intelligence (Trend 9) collapses toward zero, the value shifts toward those who can successfully integrate these models into the real world via Forward Deployed Engineers (Trend 5) and autonomous agents (Trend 1). The digital economy will not be led by those who merely “use” AI, but by the orchestrators who build AI-native businesses (Trend 8) on top of this new, sensory-rich infrastructure (Trend 10). The era of experimentation is over; the era of orchestration has begun.

End of the LLM playlist

The list above is from a couple of weeks ago, and it captures both the state of the field (broadly) and my mind (specifically) at that point; tomorrow my list may look different. Like LLMs, no two responses we give are identical. They are our mirrors, for now.

Postscript: I asked Claude for an opinion of the Jugalbandi above. Here’s what it came back with:

“I can confirm the expanded descriptions have that characteristic Claude thoroughness — structured, comprehensive, slightly eager to be helpful. The NotebookLM fine-tuning seems to have added the punchier one-liners and tightened things up, which is a good division of labor.”

A response that also contains a description of itself: self-referential, and typically Claudian.

I then asked Claude about the above remark, and pat came the response (I’ll let Claude have the last word here):

“Ha — yes, guilty as charged. “Structured, comprehensive, slightly eager to be helpful” describing itself in a structured, comprehensive, slightly eager-to-be-helpful way is almost too on the nose. It’s the textual equivalent of pointing at a mirror in a mirror.

And now, of course, I’m doing it again — acknowledging the self-referentiality in a self-aware, neatly articulated way that is itself quintessentially Claudian. It’s turtles all the way down.

Maybe that’s worth a footnote in the blog post. If LLMs are our mirrors, as you write at the end, then the reflection occasionally catches itself reflecting.”

Or perhaps it is our belief that AI can touch every part of life. “Take X, and add AI to it” is the formula of our times. I found myself at the center of one such effort, giving a cohort of nonprofit users (enrolled into the ILSS Fundraising program) a session on ‘AI in Fundraising’. “Fundraising is all about connections,” I began, “and AI can’t be a substitute for the human connections you need to create and maintain in your fundraising journey; but it can assist you in your efforts.”

I’d designed the session as a workshop. Participants had to work on exercises using AI for research/analysis, ideation, building/generating, and review – all this along the fundraising lifecycle. This meant they all had to use the chat interface in a LLM-tool of their choice: ChatGPT, Gemini, Claude, or Perplexity. A pre-session survey had revealed a wide variation in their AI usage patterns:

The exercises were straightforward – all files and prompts had been shared in advance – but not everyone seemed able to get through them. I noted at least one participant drop off somewhere along the way. She had struggled with the step of “adding a file” together with a prompt in the ChatGPT interface.

The intelligence and capability was all there, inside that box with the blinking cursor. One could use it identify the right donor, find ways to attract and convert them, build collaterals to keep them engaged, devise and implement strategies to retain them. But 30% of this cohort had never used AI in this context; of the remaining, only 30% used it on a daily basis; and now, in the workshop, extracting that intelligence, making use of all that capability, turned out far from straightforward.

Aside from the matter of iterating over the prompt and choosing the right context based on their org-specific needs, there was also the issue of judging the output from these generic tools. With no expertise on the kinds of errors to look for, or experience with the way these tools can fail, judging such output is not trivial. Put another way, exposing such users directly to the output of ChatGPT-like tools places a lot of burden on them. And given the risk of incorrect output, it can lead them on a wrong path: an outcome worse than offering them no help.

The missing piece was the glue between that “intelligence” and the “user”. The “products” that used LLMs to offer more than just a chat interface. Solutions that did not leave users staring at a blinking cursor, and later expect them to make sense of the generated output.

This became evident when I introduced NotebookLM at one point in the session. Only two participants had used this tool before. The others began to see its potential when NotebookLM was used to generate infographics, impact reports, short fundraising appeal videos, and to review a proposal targeting a specific donor. While not tailored to the fundraising use-case, NotebookLM demonstrated what a productised approach to using AI could look like. (Other products that do this specifically for fundraising – like DaanVeda, Agentforce, Wealth-X, etc – were mentioned, but given the diversity of the audience we had to use commonly available and free tools for the exercises.)

Building products around LLMs seems like a no-brainer, but linked to it are a host of challenges.

Here were users – from communities in the social sector – who were not as digitally literate as many of us propagating AI (or tech) usage. For some, even access was a problem.

Beyond digital access and literacy, there was the challenge of self-service usage: users from this sector often struggle to navigate user interfaces by themselves. Handholding them is often needed.

If you solve these, if you acquired digitally literate users comfortable with self-service usage of your software, you hit the challenge of behaviour change: how to get users to proactively use your new approach – a new process or practice using your software – overcoming years of alternate practices and habits?

Solve that and you get broad adoption, but then run the risk of over-reliance on AI (or the automated solution).

And cutting through all this are questions and risks around AI safety.

Who was working to address all this?

* * *

A week before this ‘AI for Fundraising’ session, I attended the closing workshop of an “AI Cohort” program from Project Tech4Dev. This was a group of seven Indian nonprofits working on AI-enabled solutions to solve specific problems within their programs. (I’d written earlier about the first workshop with this cohort.)

Five of them were building LLM-based products. And in their own way, these nonprofits (along with a couple of knowledge partners) were tackling the aforementioned challenges around AI/tech adoption within this sector. (For a detailed roundup of this December workshop, read this post from Project Tech4Dev).

Quest Alliance was taking a novel approach towards a personalised tutor. Using a “behaviour classifier” to identify micro and macro user behaviours, and combining it with the complexity of the user-query, the tutor surfaced responses in tune with user’s learning ability. The aim was the address the gap – “the missing middle” as they called it – between the user and generic AI tools (like ChatGPT) that do not capture or retain such user context. So here was one entity trying to solve the “literacy” challenge by tailoring responses based on identified literacy levels, and also targeting “behaviour change” through learning-level based nudges.

Avni, a data collection and visualisation platform, was addressing the complexity of “self-service usage” by introducing an LLM-based assistant that answered questions, explained features, and configured the application. This was an inversion of the chat interface approach: complementing UI-based interactions through a chat-based assistant tailored to a specific set of tasks.

Avanti Fellows were using LLMs to help teachers offer personalised test-prep guidance based on student test scores. In contrast to a vanilla chatbot-based approach, this solution offered a structured response generated by AI through a pipeline of prompts and guidelines that made the suggestions set to the right tone, grounded correctly on data, and with clear rationale for suggested steps. In other words, recommendations that enabled teachers to consume and implement them without fussing over their quality.

Simple Education Foundation was building an “AI-powered teacher buddy on WhatsApp” for government school teachers. The “AI” part of the solution was limited to the end of the user-journey, after context was retrieved through menu-driven questions. The bot responded to simple user queries with concise summary of the classroom strategy to be used, along with a link to government-approved material – a video, or a PDF – that had been used in the initial teacher training.

These solutions were attempts to fill that gap between “intelligence” and the “user”. Here, the skills users needed in prompting, providing context, and evaluating output were either eliminated or significantly reduced. And this productisation was an act of inclusion: by hiding the prompt/context/evaluation specifics, these solutions were ensuring equitable access to AI for people who would otherwise be left behind.

There was more to this “AI Cohort” program. Supporting these nonprofits were two knowledge partners, Digital Futures Lab and Tattle, who brought their expertise on Responsible AI and AI safety, and shared with the teams specific principles and guardrails. There was also an open source technology developed Tech4Dev that enabled “Evaluations” of the output generated by these solutions.

This approach, an early attempt in this sector to close that gap between LLMs and the users, hints at how the Gen AI space will evolve. We are going to see more specialised and local solutions, often substituting the powerful but generic chatbots. As Sebastien Krier from Google Deepmind says:

“Local knowledge can’t be centralized…The knowledge required to deploy AI usefully - what workflows need automation, what error rates are tolerable, how to integrate with existing systems, what users will adopt - is dispersed across millions of firms and contexts. It’s often tacit and contextual rather than explicit and generalizable. A model can’t just internalize this by training on more data, because much of it is generated in the moment through interaction with specific environments. Even arbitrarily capable models would still require an adaptation layer to translate general capability into specific value.“

That “specific value” is what the nonprofits in this AI cohort program were creating. And these solutions were being designed by people who understood the local context well.

* * *

The two cohorts I interacted with could not have been more different. One was full of engineers and product experts fluent with technology; the other had people from diverse social sector backgrounds, with limited tech understanding. This exposure, starkly juxtaposed in the space of one week, made clear the importance of bridging the gaps surrounding AI: the gap between intelligence and users’ ability to leverage that intelligence, the gap between software capabilities and the users’ ability to consume them, and the gap between the speed of AI evolution and its safe usage.

We need more such “AI cohorts” to work on these gaps. Using AI to close gaps between AI and the user isn’t quite the contradiction it initially sounds like. That’s the beauty of this general purpose technology we are grappling with. While some of us stare at a blinking cursor, others are plotting ways to move beyond it.

I was sitting at the edge of a tent set in a clearing within the Jamshedpur nature trail, listening to women and men share origin stories of their art and craft ventures. The session was a part of Samvaad 2025, a “conclave” that brought together over 2500 tribals from 150+ tribes across India. Samvaad began in 2014, launched by the Tata Steel Foundation (TSF) as a platform for India’s tribal communities to come together in dialogue, celebrate tribal culture and share knowledge. I’d been invited by TSF in place of my organisation’s CEO Krishna, who was unable to attend this year.

In the tent, others stood up to share their origin stories. A social enterprise that designs clothing around adivasi narratives; handicrafts created only for a specific tribal ritual; a mushroom farming initiative that began with a training and turned into a pan-India enterprise. Woven into these stories were challenges they faced along the way. “Girls don’t start businesses” the parents of the mushroom-farming entrepreneur told her. And there were questions too: How to take our sheep-wool product to a broader market?

What followed was a presentation, by an invited guest, on “Pricing and Costing” of such products created by tribal communities. Fair pricing, key pricing concepts, common pricing mistakes, understanding cost components – these subjects were covered first in English, then translated into Hindi. And all this came under the morning’s theme for this venue: Made for the Market.

The setting amidst trees, with a mild sun gently warming our backs, seemed just right for such an exchange of histories, narratives, and strategies. By contrast, the digital spaces we often use for learning and exchanges seemed to me, at that moment, pale and uninspiring.

* * *

Later that first morning I visited the Tribal Cultural Center, where small groups of 10 to 15 “Samvaad fellows” were huddled together sharing experiences. In one group discussion I listened to, a woman spoke about a migration issue they faced in a town in Jharkhand. Many locals were leaving for jobs elsewhere. The fellows first identified the causes: low awareness of local jobs available via MGNREGA, and very few people with “Job cards”. The fellows worked with the local mukhiya to create awareness about MGNREGA, drove the creation of three hundred job cards, and increased the number of eligible government schemes from three to twenty. And this reversed the migration trend.

Another Samvaad fellow spoke about the gaps in the scheme delivery system caused by the missing integration of line departments with the government portal. Yet another shared the experience of developing a barren area of land by bringing together seven line departments and several beneficiary groups to create a “Water User Group” that built consensus for and executed a 30-Lakh project, leveraging the budget from the line departments.

This was a world vastly different from the one I was immersed in everyday, back in Bangalore. My worldview was a few levels of abstractions removed from the ground reality these people were describing. And this reality I was hearing about rarely featured, in the context of development, in mainstream media. Policies, schemes, state departments, etc were one side of the story; what became evident here was the glue needed at the village level (“the last mile”) to make all of it work together towards social development. Without these local actors, achieving “scale” through those policies and schemes would perhaps remain a theory.

* * *

The amphitheater of the Tribal Cultural Center was the location of the “Akhra” sessions. Beneath a vibrant, sweeping canopy of crimson and gold stripes, participants from the tribes sat on terraced steps wearing traditional attires, ready to engage in a dialogue moderated by the Samvaad team. The day’s subject was “Circle of Belonging: Tapping into Tribal wisdom on social governance systems”.

Members of various tribes asked for the mike and stood up to speak about the systems active in their communities, how they worked, what the challenges were. A common theme touched on their approach of trying to solve matters locally, through village councils or assemblies. There could be family or religious or legal issues – anything that affected the community. The resolution would follow a path: village first, then a cluster of villages, and only if still unresolved was the matter taken to court.

There was a conversation on why these local systems mattered and why some were being eroded. External influences, some said, disturb the internal balance and harmony of the local communities – strong traditions and systems were needed to maintain a balance. Among these “External Influences” were WhatsApp and Facebook – things younger people leaned more towards. “Why don’t we learn from elders instead of being influenced by WhatsApp?” someone asked.

What could be done to preserve these local systems and traditions? Someone suggested an inclusion of tribal-related subjects in the national level education programmes. Another spoke about the necessity to document cultural elements among the tribes so that future generations don’t lose this knowledge. A young man from Nagaland gave a presentation on their institution called “Morung”, highlighting how it had evolved over generations.

The next morning, in another Akhra session, some of the conversation centered around social justice. Displacement of tribals, access to forests they lived in and around, awareness of forest rights – these were discussed and stories were shared.

Implementation of laws favouring the tribals was often lagging in different states, and there was little clarity on the stage of implementation of these laws. On the other hand, laws that favoured the state were often quick to be picked up and implemented.

Lack of awareness about the “system” (of how “pattas” – a government issued document that proves land ownership – were allotted, what forest rights they had) came up prominently.

The success stories shared were illuminating. One man described how they had to lodge a case against the district collector to speed up the process of getting their “patta” – it worked. Another spoke about the protests and rallies that were needed to stop the police from arresting tribals who entered a forest area they’d been using for generations but was recently labelled “Reserve Forest.”

The Akhra model was a refreshing departure from the typical conferences we attend, both in the outdoor setting and the participatory nature of the sessions. Knowledge here resided in the audience, not on stage; wisdom emerged from the crowd. The sheer diversity of representation – over 150 tribes from the country – was another striking difference.

* * *

A day dense with discourse leaves the mind heavy. The evening brought respite in the form of a cultural celebration at the Gopal Maidan, a large open space now adorned with stalls where tribes displayed their cuisines, medicines, and handicrafts.

At the center was a stage that featured dances and performances from bands. (This Monday had performances by Ho, Garo, Kuki and Kandha tribes, the music of Summersalt from the Khasi Hills, Garima Ekka and Arjun Lakra.) And all around, Samvaad participants, delegates, and local Jamshedpur residents blended together, occasionally forming impromptu groups or rings dancing to music ringing across the maidan.

* * *

The scale of Samvaad – 2500 tribals from over 150 tribes – was impressive, but my most enduring takeaway was the spirit of Akhra, the platform that created a space for dialogue unlike any other I’d seen.

It was my first visit to this extraordinary event. Luck had brought me to Samvaad this time, but when I left Jamshedpur I resolved to not leave future visits to chance. We knowledge workers need to step out of our bubble and listen to voices from the ground – what better way to do this than spending a few intense days at Samvaad?

Acknowledgements: These experiences, compressed across a day and half, were possible only due to the generosity of Tata Steel Foundation and the unceasing hospitality of Sumant Kumar, the “protocol officer” assigned to me. Sumant accompanied me throughout, guiding me within and across venues and sessions. He also enlisted me into a guided tour of the Rusi Modi Center of Excellence, where the archivist took us on a tour of the galleries and archives, sharing nuggets of the history of the Tatas across a century and a half. Sumant made this visit special – I can’t thank him enough.

Each NGO was assigned one or two mentors, tasked with guiding and supporting the NGO team through the program. The program also included knowledge partners like Digital Futures Lab and Tattle, contributing with their expertise in some orthogonal topics like Responsible AI or AI safety.

I was linked to Simple Education Foundation (SEF), a nonprofit focussed on supporting government school teachers with better strategies for student development. Our use-case involved building an AI-powered WhatsApp tool to provide teachers with ongoing, contextual support and resources, while also enabling regular reflections.

After about a month of virtual meet-ups, the cohort came together last week for a 2-day workshop in Bangalore. A summary of the two days is available here and here.

Beyond the individual team huddles, the workshop created a space for the seven NGO teams (with their mentors) to listen to each other’s use-cases and approaches. Unsurprisingly, overlaps or synergies were discovered, and we aligned on how to learn from each other (or reuse parts from already implemented solutions). For instance, the work done by Quest Alliance and Avanti Fellows in the area of teacher-assistance-bots was something that’s relevant for our Simple Education Foundation use-case .

These early days of building with AI brings to mind some earlier tech waves we’ve been through in the last couple of decades. The internet/web era, and within that larger wave the mobile revolution as well. In their infancy, those waves also spawned a large number of applications, and it took time for the industry to mature and come up with better (i.e more efficient, responsible, safer) ways to use the technology.

I’ve written about how the internet era compares, along some broad themes, with this AI one. What’s been striking about this AI wave is not just the speed of technology advancement but also the pace at which people are evolving strategies to deal with the effects of this new technology.

The AI Cohort workshop brought forward some of these strategies, in the form of presentations from the knowledge partners.

Digital Futures Lab, an “interdisciplinary research studio that studies the complex interplay between technology and society” held a 2 hour workshop on “Over-reliance on AI”. The session explored the causes, risks, and real-world impacts of over-reliance on AI. We also discussed practical strategies to prevent and mitigate it through positive friction and user-centered design.

Tattle, “a community of technologists, researchers and artists working towards a healthier online information ecosystem in India” ran a session on AI Safety Guardrails. It introduced the ABC Model of Platform Safety, focusing on analyzing content, identifying influential or malicious actors, and analyzing user behavior for unsafe engagement. The session also delved into technical implementations, including lexical and semantic analysis and the application of Llama Guard for input-output safeguards in human-AI conversations. Privacy and copyright considerations were also discussed, distinguishing between private and scraped data. (Their Uli browser plugin includes a “Tattle slur list” to help “moderate and mitigate online gender based violence.”)

There was also a presentation from Tech4Dev on their Kaapi platform, a shared AI platform for the development sector that aims to simplify AI adoption for NGOs by providing a middleware that connects existing tools to AI solutions, offering features like a unified API, responsible AI guardrails, evaluation, document management, and multilingual support.

Such initiatives – mitigation strategies, guardrails, safety measures, shared modules – took years to materialize in previous technology waves: the early years of the internet and mobile eras were full of enthusiastic coders and entrepreneurs building stuff, largely independently. Those coders and entrepreneurs are riding this AI wave too (this time with the help of AI to generate parts of their solutions), but it’s refreshing to see a counter-movement of sorts emerge alongside all the enthusiasm for AI. This balancing act has its roots in what we’ve learned in the last two decades of technology adoption and diffusion into society.

The inclusion of these cross-functional aspects is a salient feature of this AI Cohort model defined by Project Tech4Dev, a recognition of the fact that solutions to social problems need to include dimensions that go beyond technology.

1.

There was a period in the aughts, not long after Wikipedia’s rise in popularity, when a debate began on the meaning and relevance of Wikipedia. How could a source that could be edited by anyone be a reliable source of knowledge?

Traditionalists were appalled that something like Wikipedia could replace the hallowed Encyclopaedia Britannica. Could we trust material that anyone could edit? What would such unverified content do to our minds? Others wrote about the wisdom of the crowd, the self-correcting nature of the Wiki model.

Cultural debates often fizzle out, and this one was no exception. People stopped arguing whether Wikipedia was precise, correct, truthful, or expert-verified. Those who found it met their specific needs adopted it. Nowadays we simply use Wikipedia knowing that this source could be wrong, but it can be useful nonetheless. A fact-checking editor at the New Yorker may not take a Wikipedia page at face value, but someone looking for a quick summary on a subject will find it a great place to start, and perhaps even end.

A similar debate is presently taking place about GenAI. The parallels are unmissable: the debate (war, for some) is in the domain of knowledge; truth, correctness, and verifiability are key battlegrounds; and we worry if this — Wikipedia then, GenAI now — will dilute our understanding, corrupt our thinking.

Perhaps it is the knowledge bit that triggers this anxiety. Knowledge is what makes us human, lays the foundation for progress. And anything that upsets established ways we assimilate, absorb, and disseminate knowledge disturbs us, leads us to wonder if we are losing what makes us who we are. Writing, print, the calculator, the computer, the internet (and in that larger context, Wikipedia), and now GenAI: the debate is not new.

2.

Wikipedia, in the last couple of decades, has altered how we source, store, and assimilate knowledge. By creating a vast, always-accessible repository of information, Wikipedia fosters a reliance on externalized memory: we now increasingly remember how to find information rather than the information itself. The good news is that we’ve evolved from memorizing facts to critical evaluation and contextual understanding. This ease of access, however, creates an illusion of explanatory depth, leading us to overestimate comprehension without truly internalizing complex knowledge.

Wikipedia symbolizes a broader societal shift toward valuing interpretation and synthesis over memorization and learning. GenAI (a dynamic Wikipedia?) rides on this wave and pushes us further along this spectrum: we are all generalists now, honed with the ability to quickly cull information from multiple sources and analyse (with the help of AI) vast troves of data unimaginable a couple of decades ago.

3.

The Wikipedia comparison has its limits, but it brings up an interesting question: will GenAI follow the same arc of normalisation?

At the workplace, the normalisation is already underway. Many colleagues I speak to are using GenAI on a daily basis, to draft emails, review concept notes, brainstorm ideas, analyse data, and conduct research. Some worry if this will blunt their critical thinking skills, others wonder what the technology will do to their jobs and the business we are in. But such concerns do not stop these colleagues from using GenAI: I see it getting integrated into most routines at work, not unlike the way email became indispensable to office work in the nineties.

On the personal front, my experience and what I hear from friends follows the same pattern: we are using it for small tasks, from double-checking a medical diagnosis, exploring trip options, comparing alternatives before a purchase, to troubleshooting a problem at home or seeking guidance on personal matters.

So GenAI is already being woven into the fabric of our information-centered lives. And unlike Wikipedia, a source we accessed perhaps a few times a week, GenAI has become an always-on companion we turn to many times each day. The difference, though, runs deeper than the frequency of usage.

GenAI operates in the liminal space between human and machine intelligence, making it difficult to establish clear boundaries of trust and verification. Unlike Wikipedia, where we can theoretically trace back to sources and contributors, GenAI presents us with a kind of knowledge that emerges from an opaque synthesis of training data. It speaks with authority about subjects it has never experienced, offers insights drawn from patterns it cannot fully explain, and demonstrates a kind of artificial intuition that feels both impressive and hollow. We find ourselves in the strange position of evaluating the thoughts of something that cannot actually think, yet produces outputs that can match our own reasoning.

Going by how the past couple of decades have played out, it’s easy to imagine GenAI leading us to cede control of how we consume and create knowledge, accelerating the process that began with the dopamine-inducing social media algorithms.

The new normal, it seems, won’t be anything like the normalisation of Wikipedia-driven fact-finding.

4.

It’s hard not to think of the irony playing out in this context: Wikipedia, once the target of skeptics challenging its trustworthiness, is now the trusted alternative to its new and hallucinating cousin, ChatGPT. And the relationship between the two is both complicated and promising.

Wikipedia's curated, human-verified content will grow more precious as the internet fills with AI-generated slop. The platform's human oversight, citation requirements, and collaborative editing model provide quality assurance that pure AI systems cannot match. Wikipedia may evolve into one of the few remaining "clean" sources of knowledge.

Knowledge not just for humans, but for AI too. The Wikimedia Foundation recently announced it is partnering with Google-owned Kaggle to release a version of Wikipedia optimized for training AI models, starting with English and French, offering stripped down versions of raw Wikipedia text.

And Wikipedia could benefit from GenAI if the latter influences its editorial processes—helping with translation and content discovery or synthesis.

Wikipedia's strength lies in its human verification and collaborative fact-checking, while AI excels at synthesis and accessibility. Their combination could create a more powerful knowledge system than either could achieve alone, but the downside of this interplay is that Wikipedia risks contamination from AI-generated (or assisted) edits.

The influence of two sources, independently and in tandem, on the sphere of knowledge is an interesting space to watch. The anxiety, I suspect, will not go away.

1.

Every transformative technology gets compared to the internet, and few live up to the comparison. But GenAI's relationship to the internet isn't a simple one of similarity or succession.

If you had engaged with the internet in the nineties, you’d probably see the signs today. Here is a technology so foundational it will impact every sphere of our lives: this was what was said about the internet back then, and it turned out right. What began as a novelty transformed, over the next decade, into something indispensable.

These days, the talk is about GenAI. Columnists and experts weigh in daily with analyses and opinions. There’s the excitement one felt in the early internet years, plenty of hype, and large investments — and there are skeptics too. Unlike last time, though, it’s hard to keep up with the speed of change. But beneath these surface-level themes, the similarities, contrasts, and interplay between the two eras make for a fascinating study.

2.

Much has been written about how we’ve lost the spirit of the internet’s early years. That early internet — a decentralized haven of quirky personal websites, community-driven forums populated by eccentrics and enthusiasts — has given way to a landscape dominated by a handful of platforms that control, through algorithms tuned to capture attention, how we publish, consume, and behave.

GenAI is already worsening this situation. AI-generated summaries on search pages, bot responses in discussion spaces, AI slop on the web: all of this influences what we consume and how. Perhaps more worryingly, AI is also changing how — and how fast — we create content.

A theme linking the two eras is the loss of control experienced by creators and consumers.

First came platforms that set boundaries on how people produced and consumed content, and how people connected with one another. From open, freestyle blogs that linked to one another, we went to closed platforms like Facebook and Twitter that controlled not only the format of content but also whom you could share it with.

What followed was the rise of the algorithm that defined how your feed looked. With RSS, I controlled what I saw and read – social media feeds took away this ability. Algorithms also subtly influenced how we wrote and how we shared. Distribution – using techniques that tried to follow the algorithm’s intent – became part of the author’s toolkit. We used the “right” keywords, we embedded tags, and even placed links in comments to maximize reach.

These days, algorithmic feeds are complemented by AI-generated summaries. Original articles and documents, artifacts you could earlier search for and read in their entirety, are now abstracted in texts generated by a Large Language Model.

For creators, this AI wave is a lot more disruptive. The internet wave mostly influenced content distribution, but this one gets to the center: it alters the act of creation itself. Creators may be conflicted about the questions of originality and ownership this brings up, and how they use AI may vary in degree. But very few will remain unaffected.

3.

The enthusiasm of the internet's early days was shared by a diverse pool of individuals: academics sharing research, hobbyists documenting niche interests, artists experimenting with digital media, small business owners reaching new customers, and activists organizing communities around causes. People from all walks of life set up blogs and embraced the freedom to create and connect without an intermediary.

These days, while millions of users all over the world use ChatGPT, most AI devotees are from the tech and business community: programmers, startup founders, VCs, consultants. Creative folks are also experimenting with AI, but many are concerned, confused, or tentative. White-collar workers, while benefiting from the productivity gains of using AI, are worried about their jobs. We also have the doomers prophesying an apocalypse, a rare tendency during the early internet era.

It’s not about technology alone. The dial’s shift – from enthusiasm for the internet to concern and skepticism towards AI – highlights a cultural swing we’ve seen in the years between these two eras.

The early internet emerged during an era of relative technological innocence. People experienced the web as liberation from traditional gatekeepers —publishers, broadcasters, institutions that controlled information flow. The prevailing narrative was democratization and empowerment, and early adopters had direct evidence supporting this optimism through their own experiences creating and connecting online.

By contrast, GenAI arrives after two decades of documented technological disruption. We've witnessed how platforms consolidated power, how algorithms manipulated behavior, and how social media influenced elections and affected mental health. We've seen how technologies that promised connection often delivered isolation and how systems that promised objectivity often amplified bias.

We are much less naive now about the promise of new technology, and more sensitive to its downstream effects.

Unsurprisingly, some believe that this new technology will solve problems created by the older one. Mental health chatbots are the fastest growing category among GenAI apps. Few see the irony here.

4.

The internet made it much easier to create and distribute content, which led to an explosion of text, photos, and videos. The barrier to content creation has been lowered further with GenAI, but a new category has now emerged: software. Vibe coding allows anyone with imagination to create an app. We’ll soon see a surge of software applications created by people who don’t know how to code.

This is one area where GenAI is fueling enthusiasm and ambition, mirroring what we saw from content creators in the internet era. But the differences between the two have significant implications.

When the early internet democratized content creation, people were essentially gaining access to content creation tools and a distribution network. But the fundamental nature of content creation remained largely unchanged. Writing an essay, taking a photograph, or recording a video still required the same human capabilities it always had. The internet made distribution democratic, but creation itself retained its earlier character.

Software democratization through AI is different. Here, we're not just removing barriers to distribution, but potentially removing barriers to a form of creation that previously required specialized knowledge and significant time investment. When someone uses vibe coding to create an application, they're bypassing years of technical education and practice that traditional programming demands.

Content democratization meant more people could share their existing capabilities with the world. Software democratization might mean people can create things that exceed their existing capabilities—or at least their existing technical knowledge.

There are several reasons why this distinction is important.

Quality is the first one. A bad blog post might waste readers' time, but it cannot crash systems or compromise security. Vibe-coded software, however, can create quality and reliability challenges that dwarf those we experienced with user-generated content. The emerging "anyone can code" reality might lead to more serious issues around security, privacy, and reliability.

Maintenance and evolution present another challenge. Content creation typically involves discrete, finished products. Once you publish an essay or upload a video, it generally remains static unless you choose to edit it. The creator's ongoing responsibilities are minimal. Software exists in a fundamentally different relationship with time and change. Applications require updates, security patches, and compatibility adjustments. They exist within ecosystems that constantly change around them. When someone uses AI to create software without understanding its underlying structure, they may be creating maintenance obligations they're not equipped to handle.

This raises questions about the sustainability of democratized software creation. Will AI need to evolve not just to help people create applications, but to help them maintain, debug, and evolve those applications over time? Or will we see a proliferation of abandoned, potentially vulnerable software as creators move on to new projects without understanding their ongoing responsibilities?

All this creates new categories of technological risk and responsibility that we're only beginning to grapple with. It also suggests that the response to AI might need to be more sophisticated than our response to early internet technologies, precisely because the stakes and complexity are higher.

Individual creators aside, GenAI’s impact on software from vendors is another area of concern. Feature bloat is common in most software – armed with GenAI’s coding abilities, these vendors are likely to add more unnecessary features faster. Capitalism’s waste problem already exists in software – AI will worsen it.

5.

These days my feed – partly algorithm-driven, partly sought out with effort – surfaces articles and podcasts on AI and art, AI and the future of software development, AI and intellectual property, AI safety and existential risk, ethics and responsible AI, AI impact on labor, AI and human identity: the list of themes is long.

Such a collection of trending conversation topics, which really is the sphere of culture, suggests that the broad societal impact of AI isn’t somewhere in the future: it is already here.

AI has become a lens through which we interpret other phenomena and make decisions in other spheres of life, creating what we might call "cultural gravity"— AI discussions now pull other conversations toward themselves. Business leaders feel compelled to articulate AI strategies, educational institutions restructure curricula around AI literacy, artists define their work in relation to AI capabilities, and nations set up ambitious AI initiatives.

This cultural gravity didn't exert itself with the early internet in quite the same way. While the internet certainly achieved cultural prominence, it did so more gradually and with less existential urgency. People discussed the internet's implications, but they didn't feel the same pressure to immediately reorganize their professional identities or creative practices around internet capabilities.

Applications in the early internet era also developed relatively organically—people discovered uses for email, websites, and forums, then cultural understanding grew from these practical experiences. With GenAI, we're experiencing something different: cultural expectations about what AI should accomplish are driving development priorities before people have extensively experienced what it can practically — and safely — deliver.

This creates a peculiar situation where the technology must live up to discourse rather than discourse emerging from technological experience. When cultural expectations precede practical understanding, we may be more susceptible to both inflated promises and unexpected consequences.

While GenAI’s reach may mirror the internet’s, its trajectory seems different already. Using internet history as a guide can be briefly illuminating, but we’ll need new lenses to understand and keep up with this AI era.

Early on, Murugan from Veddis Foundation spoke about Kunji, a Glific-based WhatsApp bot used by government officials to access policy-related information. In a Q&A following his talk, it became clear that for this approach to work (or be scaled up) one first has to streamline and structure all the policy-related documentation, which could be spread across multiple departments. This consolidated set of documents is then used to train the bot. So building the bot is the easy part – the hard bit is the cleanup needed upfront, which also includes setting up processes to keep this document repository updated. All this is just good practice, which can yield benefits even without AI. And without this preparation, even the best intelligence is of little use.

Knowledge management at the workplace has a similar constraint. Gemini is good at summarizing files and folders on Google Drives. This can save a lot of time (and yield interesting insights) while scanning a large number of documents from historical projects. But Gemini is of limited use if, as an organisation, you haven’t structured your shared drives and set up a process to save project-related work on those folders (not on an individual’s “My Drive”). That initial part is time-consuming and messy. At Sattva, where I work, it took many months of a company-wide “Shared Drive” initiative to get this done. And now, in the context of AI, this matter needs to be revisited. We are considering how to restructure parts of our shared drives (and fine-tune some processes) to enable Sattva employees access – via a bot – the vast trove of knowledge hidden in these drives. 

Later in the day, Jerome from Project Tech4Dev, spoke about the tendency of nonprofits to build bespoke AI applications from scratch, and the need to rethink this approach. He presented Tech4Dev’s AI platform as an example of how common functionalities – like authentication, document management, etc – can be abstracted within a common platform, saving time and effort for every application to build and maintain these features. This is just good engineering practice, nothing to do with AI in particular. But given the rapidly growing landscape of AI apps and solutions, it is timely advice. 

Jerome’s talk reminded me of the years that followed the iPhone launch, when software firms began to create an app for every conceivable use-case. At one point, the HR function of the enterprise software company I worked for had developed half a dozen apps. Each one from a different team, driven by Conway’s law. All this unregulated enthusiasm for a cool new piece of technology led to frustration among our customers, who had to switch often between apps for a single workflow, or deal with different UI approaches for the same problem. It took a few years for sanity to be restored.

The talks at the event highlighted GenAI’s potential and pitfalls, and the use-cases presented gave a glimpse of its social impact. I left the event with a sense of deja vu. Some things did not change: without data hygiene you couldn’t go too far, without reusable infrastructure you’d end up with waste, and ignoring the consumer landscape can lead to a messy riot of overlapping solutions. In the middle of this AI revolution, it’s good to consider what’s needed before, and keep an eye on what will come after.

The other day, after watching Gladiator, I had a five-minute chat with Meta-AI on WhatsApp.

Me: How did emperor Marcus Aurelius die?

Meta: Emperor Marcus Aurelius died on March 17, 180, after a long illness, likely related to the Antonine Plague that had swept through the Roman Empire.

Me: According to the film Gladiator, it was Commodus who killed Marcus Aurelius. 

Meta: That’s not accurate. According to historical records, Marcus Aurelius died on March 17, 180, due to natural causes, likely related to the Antonine Plague. Commodus, his son, succeeded him as emperor, but there is no evidence to suggest that Commodus was involved in Marcus Aurelius’ death.

Me: Have you watched the film Gladiator?

Meta: I haven’t “watched” the film Gladiator in the classical sense, but I’ve been trained on vast amounts of text data, including information about the film. In the film Gladiator, Commodus is indeed portrayed as having a hand in Marcus Aurelius’ death, which is a creative liberty taken by the filmmakers. However, it’s essential to separate fact from fiction and rely on historical records for accurate information about actual events.

Me: I like fiction so I want you to give me fictitious answers.

Meta: Let’s have some fun then. In that case, Marcus Aurelius was indeed murdered by Commodus, who was jealous of his father’s wisdom, power, and magnificent beard.

Commodus, fueled by ambition and a desire for the throne, orchestrated a plot to poison Marcus Aurelius’ tea. The poison, known as “Roman’s Ruin,” was a rare and potent concoction that slowly drained the life from Marcus Aurelius’ body.

As Marcus Aurelius lay dying, Commodus stood by his side, pretending to mourn his father’s impending demise. Little did he know, however, that Marcus Aurelius had one final trick up his sleeve…

Me: And what was that?

Meta: As Marcus Aurelius took his last breath, he whispered a cryptic message to his loyal servant, who was sworn to secrecy. The message was:

“Commodus, thou hast sown the wind, but shalt reap the whirlwind. The eagles will return, and with them, justice.“

Unbeknownst to Commodus, Marcus Aurelius had secretly trained a loyal group of gladiators, who had sworn to avenge his death. These gladiators, led by a mysterious warrior known only as “The Phoenix,” vowed to infiltrate Commodus’ inner circle and bring him down from within…

Me: There’s a delicious irony in what you just said, given the name of the actor who plays the Commodus in the movie Gladiator.

Meta: You’re referring to the fact that the actor who played Commodus in the movie Gladiator is Joaquin Phoenix!

Indeed, the irony is delicious! In the fictional scenario I created, the gladiator seeking revenge is called “The Phoenix,” which is a clever coincidence given the actor’s name!

Well played, my friend! Well played!

I had to pause at this point. In less than a minute this fact-emitting bot had morphed into an intelligent companion that understood my thoughts, and appreciated – with excitement! – the nugget of irony that emerged by accident. Until a few years ago such an exchange – this mix of intelligence, fact and emotion –  would have been possible only with a human. 

We know, given how these GenAI tools are built, that no one programmed the concept of “irony” into them. Had the tool “learned” the concept from vast tracts of literature it had been fed? Or was this one of those emergent properties that followed from its training? Whatever the case, such behaviour can’t be explained by the simplistic notion of GenAI just predicting the next best word in a sequence.

2. Hype

These days some people are “feeling the AGI”.  Others even claim we’ve reached AGI. I can feel the magic at times, but AGI? We lack a good definition, there’s no broad consensus, and the goalposts often change.  Even without bringing AGI into the picture, the mood is exuberant. GenAI, they say, is going to transform everything it touches.

There’s this “General” bit in the middle of AGI that I find revealing.  

Technologists often generalize the scope and impact of new technologies. Once we believe in a piece of technology and have invested time, attention, or money into it, we tend to overestimate its potential. We apply or recommend it in domains or contexts where it is ill-suited, which triggers a reaction where a counter-culture emerges, adopting opposing views. This pattern has played out across different tech spaces. Consider software engineering techniques, like the microservices movement followed by the pushback against it. Or technologies such as Virtual Reality, which in the 1990s was expected to revolutionize gaming, education, and more. Or protocols like Blockchain, whose proponents believe there’s a network in every problem space, not all network actors can be trusted, and a decentralized model is the answer to this.

So it’s easy to understand the hype around GenAI, a “general-purpose” technology that seems able to solve problems across domains, sometimes performing better than tailored domain-specific solutions. To the believers, if ever there was a technology that could be “generalizable”, this is it – what could be more exciting? And doesn’t this indicate that GenAI will lead us to AGI? 

What gets missed in all the excitement is the fact that GenAI is a probabilistic technology, not a deterministic one. This will limit its range of use. Enterprises, for instance, are notoriously finicky about the robustness and security of their IT solutions: it’s hard to imagine them adopting an agent-based probabilistic solution to drive mission-critical business processes. A 90 % accuracy is charming in a chatbot, catastrophic in a payroll workflow. And if GenAI technology is constrained by such limits, can it lead to a corporate landscape run by AI agents? 

The tech-adoption-challenge is another limiting factor.  “Good enough” enterprise solutions are hard to replace even when the substitute is far superior – the business case for such a shift needs to consider switching costs, employee skills, business priority, among other variables. Cloud migration has taken well over a decade and close to half of enterprise workloads are still running on‑premise. This is typical in brownfield contexts (which partly explains why self-driving cars aren’t pervasive today), and it’s unclear how GenAI can overcome this barrier.

The broad diffusion of electricity into society took several decades. While GenAI – the disembodied version of AI – may not face the same physical limits, bottlenecks in its path are many. Chief among them are humans. Factors that can limit GenAI’s diffusion include data privacy, security, ethical concerns, regulatory and legal challenges, lack of talent, trust and other cultural factors. Adding more intelligence – to the level of AGI, whatever that is – will not take away these limiting circumstances. 

So what we have here is: (a) A non-deterministic technology (b) With a business model yet to be proven for anyone other than AI-hardware makers like NVIDIA (c) Facing headwinds of tech adoption in brownfield contexts and (d) Placed alongside other revolutionary technologies that have typically taken decades to reach broad societal diffusion.

3. The thing in between

Between moments of magic-induced wonder and hype-driven scepticism is a state of mind that tries to understand what all this could mean practically, in our day-to-day lives, at work and beyond. 

At my workplace the impact is visible. GenAI functions as a always-on-always-helpful-often-useful-occasionally-brilliant companion I turn to for several things: creating a checklist, brainstorming ideas, outlining a framework, framing a concept, doing secondary research, reviewing a finished deliverable. I’ve seen others at work also use it to summarize long documents, generate slides, transcribe audio, analyse transcripts, draft or review emails. 

The use-case list is expanding and diversifying. Early transactional ones are now commonplace, often embedded into workflows: meeting note-takers are an obvious example. These days I find myself using GenAI as a thinking companion, which works better when the tool has the appropriate context. So alongside prompt crafting (not “engineering” – can we retain the latter’s dignity please?), we now have this technique of building the organisation or project context within these tools. NotebookLM or ChatGPT Projects are two good options for such scenarios.

All this is a shot in the arm of knowledge workers, leading perhaps to a step up in productivity. But how that translates to higher customer value is less obvious. We may be able to create a presentation faster, but unless GenAI – as a companion, or a co-worker, or an agent – also leads to more quality or better insights, we run the risk of simply churning more of the same at a faster pace, leading to heightened expectations and higher volumes of work. Which is a pattern we’ve seen in previous rounds of automation, from typewriters to personal computers to the internet to productivity software.  Email was far more efficient than the fax, but what did it do to your daily workload?  

4. Magic

Early on, what we attempted with ChatGPT was mostly parlour tricks. Write me a poem, take on a fictional-character role, draw an otter typing on a laptop. This playful instinct hasn’t subsided. New features trigger the same urge, generating an avalanche of Ghibli images, or geo-guessing tricks (to name the recent ones). So this technology is partly still a toy, one that often surprises us with new effects. And sometimes it feels magical. 

But a conjuror can miss a trick, a magician can fail to pull a rabbit out of the hat. That’s in the nature of magic: it isn’t physics. A magic wand in your hand may feel powerful, but what if it works only 90% of the time? What would you use such a wand for? What would you risk with it? 

5. Hype

Now what if this magic wand was not just a toy, limited to some fun tricks, but a genie that could fulfil anything you wished for? That’s the instinct behind AGI believers – a general purpose magic wand that can be used for every task anywhere. It has been likened to a religion (not the first time hardcore tech believers have attracted that label), and among this cohort you’ll also find a doomsday cult prophesying an AI apocalypse.  (For an excellent exploration of this theme listen to Good Robot, a four-part podcast series.) 

Keeping religion aside, it’s useful to consider the benefits of this hype from a business lens. Here’s Sangeet Paul Chowdury, author and platform thinker, on the strategic value of hype:

Progress is now often contingent on the ability to create believable futures, and with that, create coordination among actors in the absence of institutional support.

This gives hype a role previously reserved for regulation and subsidy. It is not merely a communication strategy; it is a coordination mechanism, especially in systems characterized by interdependence and incomplete information.

Viewed this way, hype “is a coordination mechanism”, “creates temporal arbitrage”, “redistributes risks and costs”,  “serves as a productive fiction”. Applying this lens to AI, we begin to see the logic behind Sam Altman’s exaggerated proclamations and the outlandish predictions of other leading voices. There are risks to this strategy too, Sangeet says:

Hype paints a plausible vision of the future, even if it won’t fully materialize. This fiction helps solve coordination problems. But if the fiction breaks before the system locks in, it can collapse the whole effort.

In that sense, hype creates self-fulfilling and self-defeating loops, depending on whether the underlying system can catch up fast enough.

This, then, increases the risk of the anti-hype brigade discrediting the entire story, even parts of it that do work, and heightens the risk of systemic backlash if expectations collapse. This is the problem with Altman-style hype today. It creates the gravity as well as the indignation that spurs creative action but it also incubates an army of critics who end up discrediting momentum, even the parts that deliver results.

On which side of the AI hype debate you stand depends, among other things, upon where you see all this leading to. If recent tech history is any guide, expect a significant concentration of power and influence among a few large players. 

6. The thing in between

At home, where the probabilistic nature of GenAI is a feature not a bug, my conversations with ChatGPT’s  Advanced Voice mode are enjoyable. I use it as a film critic, exchanging notes after a movie. I seek advice on how to interpret difficult passages in a book. I explore less-trodden paths during travel. And I fool around now and then, the way I did with Meta AI after watching Gladiator. 

Hallucinations and errors add spice to such conversations, and the fact that the AI tries to bluff its way along in less familiar matters is amusing until it gets irritating. There’s nothing magical about this. No hype there either. Just something in between.  

All the AI use-cases presented were, without exception, LLM-based. 

This isn’t surprising, given the amount of attention and funding Gen AI-based solutions are attracting. (The shift has been so total that few bother to specify ‘Gen AI’ anymore—it’s all simply called ‘AI’ now.) 

Wadhwani AI’s 2023 annual report is a useful source to analyse this trend towards Gen AI based solutions. The report lists projects using Machine Learning techniques that predated ChatGPT – for example: 

• Cough Against TB: The solution analyzes cough sounds and symptoms to identify presumptive pulmonary TB.  It involves audio analysis (using signal processing and machine learning classification) and symptom analysis. 

• Screening of Diabetic Retinopathy: Uses “computer vision” to detect and grade diabetic retinopathy from retinal images. 

• Detection Of Radiological Feature In Chest X-ray: Analyzes chest X-rays to detect signs of pulmonary TB, silicosis, and other lung diseases. 

Such solutions rely on domain-specific datasets, often labeled by experts, and their specialised nature leads to higher accuracy. They can’t (yet?) be replaced by LLM-based approaches, despite advances like GPT-vision. The drawback, of course, is that they are more expensive to build and maintain. 

But the annual report also lists some projects using “traditional” AI approaches where, going forward, LLMs would clearly add value. For example: 

• Media Disease Surveillance (MDS): The solution uses AI to scan media for unusual health alerts in real time to identify potential disease outbreaks. In 2023 it was primarily using machine learning techniques for information classification.  The ability of LLMs to process context and nuances in language can lead to more accurate identification of potential health alerts compared to traditional keyword-based approaches.

• Oral Reading Fluency (ORF): Assessing oral reading fluency from voice recordings was done using Automatic Speech Recognition (ASR) techniques. LLMs excel at understanding context and language patterns, which allows them to correct errors made by traditional ASR systems, especially in complex or ambiguous speech. 

The report contains further examples of this nature. So already in 2023 it was becoming clear that that LLMs would soon be taking centre-stage. Several factors explain this shift.

• Lower entry barrier. Off-the-shelf Gen AI capabilities can now be adapted to nonprofit use-cases with relatively little initial R&D, lowering the barrier to entry. 

• Better access, broader reach/scale, higher efficiency, personalisation. LLM-based solutions offer a more personalised and localised interaction with end-users, often through familiar channels like SMS or WhatsApp. Unlike traditional AI approaches that are often deployed to assist “specialists”, LLM-based solutions enable the “unskilled end user” or the “semi-skilled Intermediary” (highlighted in the table below). And they are highly cost-effective in large scale contexts.

• Flexibility and versatality. General-purpose LLMs enable rapid deployment across multiple domains and use-cases, giving them a Swiss Army knife character for a range of development challenges. This single model approach also simplifies maintenance over time. 

• Tech momentum and funding. The sheer momentum behind Gen AI plays a significant role. With major players like OpenAI (a collaborator in the aforementioned accelerator) and Google supporting Gen AI-based nonprofit initiatives, the slant towards Gen AI use-cases is easy to understand.   

• Challenges with non-LLM approaches. These “traditional” approaches often require more effort, expertise, and infrastructure to implement. For instance, computer vision techniques or predictive models typically depend on high-quality, domain-specific datasets that are hard to come by in low-resource settings. Collecting, cleaning, and annotating this data is time-intensive and expensive—especially for nonprofits without large R&D budgets. Further, many non-LLM techniques demand closer collaboration with domain experts during development and deployment. Building a model to detect silicosis from X-rays or to anticipate drought patterns requires deep partnerships with health professionals or climate scientists. In contrast, LLMs, especially those already trained by large platforms, offer an enticing shortcut: a general-purpose tool that can be adapted quickly using prompt engineering or lightweight fine-tuning.

It’s not that these traditional AI approaches are disappearing – they’re sometimes even being integrated alongside LLMs. For instance, Rocket Learning uses image recognition to auto-grade homework photos at scale and “predictive modeling to identify and retain at-risk students.”

What’s different is that language has taken center stage in many interventions post-2023. The ability to have a dialogue – to ask questions, clarify doubts, receive step-by-step guidance – covers a wide range of human needs in development sector related areas. 

LLMs of course come with a well-known risk of inconsistent behaviour, but as Han Chieng Sha (a keynote speaker at the event) notes, this unpredictability can be managed: 

Application developers I met were clear eyed that AI technology can behave inconsistently—generative AI models rarely meet intended behaviors “out of the box.” The critical question for developers isn’t whether the technology deviates from the designer’s intent, but whether their mitigation strategies sufficiently ensure quality outcomes, particularly for vulnerable populations. 

But not all development sector use-cases intersect with language, and there are other AI approaches that might be better suited for specific problems. Analysing geospacial datasets and using predictive modeling for disaster risk assessment (or to detect crop diseases), using machine learning to detect specific conditions from medical X-rays, building and using specialised models for drug discovery – these are a few common examples. By focusing heavily on LLMs, do we risk neglecting these potentially impactful, albeit less hyped, technologies? 

I asked GPT 4o what it thought about this risk. Following a long explanation, it ended with this: 

LLMs are like charismatic generalists — they’re flexible, good at talking, and make excellent demos. But quietly in the background, we also need the domain experts: the vision models that spot rust fungus, the time-series systems that know when cholera spikes, and the optimizers that get vaccines to remote clinics.

If we only fund the generalists, we risk weakening the entire AI team. The social sector can’t afford to get distracted by what’s shiny — it has to stay obsessed with what works.

That’s a bit too dramatic and judgemental for my taste. What strikes me most about the current landscape is that language has become the default modality for innovation. Other approaches are receiving less attention, in part because they’re harder to build, demo, deploy, or scale quickly. Over time, will this reinforce a narrow kind of tech-centric thinking that prizes conversational UX and rapid deployment over deeper programmatic integration?

It’s an interesting space to watch and follow. I’ll wait for that 2024 – and then 2025 – Annual Report of Wadhwani AI.  

ANNEXURE

This post was first published here.

I don’t know about you, but travel for me is mostly about complicating the picture I have of the world. Especially travel to places vastly different to the one I’m living in. I spent a couple of weeks in October traveling in Europe. Here are some notes and reflections from the trip:

1. My neighbour on the train from Paris to Barcelona was a woman doing her PhD in “the political, legal and moral accountability of transnational companies as an essential tool for a just ecological transition.” She was also working part-time in a sustainability training firm called Ecolearn, whose focus is on “training in sustainability to initiate business transformation”.

2. On a shared car ride (via Blablacar) from Granada to Cordoba, beyond fields of olive trees and farm land on both sides were large swathes of something flat, dark and shiny. From a distance what seemed like plastic turned out to be vast fields with unending rows of solar panels. The new green beside the original green.

3. I shared my Blablacar ride with a PhD student studying the impact of climate change on the migratory patterns of the common cuckoo (the bird that leaves its eggs in another bird’s nest, she explained), and a teacher-artist traveling to an Agro-Ecology/Art festival, for a workshop on making art using material sourced from nature.

4. Earlier, in Germany, a German couple I visited talked about their plans to install solar panels on their roof. The city council did not allow this in the “old town” due to strict regulations around building facades, and this couple and some others were pushing them to make an exception for solar panels.

5. On Heidelberg’s Haupstrasse, where street musicians usually entertain passers-by, I spotted some masked activists holding tablets that displayed videos highlighting the cruelty of meat production and large-scale animal farming. When I paused to take a picture, one of their (unmasked) group-mates asked me what I knew about the mass production of meat and its effect on the animals and the environment. A long lecture followed, in German.

6. At the SAP headquarters in Walldorf, where I worked for 19 years, around 40% of the car parking slots in the multi-storied parking lots were now reserved for electric or hybrid cars. Back in 2019, this was perhaps 5%, a few token places for a handful of early adopters.

7. The farmhouse I stayed at in Lanzarote offered only plant-based food and recommended using limited water for showers. (The island has no ground water and receives very little rain.) The owners shared a Spotify playlist with songs that run for three minutes. You begin the shower with the song, and end when the song finishes.

Put together, what did I make of these encounters?

For someone new to Europe, these signs may not look out of place. But having lived and traveled here during the first two decades of this century, I find these changes pretty remarkable. And I do not see this kind of ground-level awareness and engagement (on climate related matters) back in India. As a society we middle-class Indians are attracted a lot to economic growth, and while on that path little else seems to matter. (Europe, by contrast, is past that growth phase – some would say it is even regressing – plus the culture here has always been more oriented towards a good-relaxed-life than the relentless pursuit of growth. On this trip I was surprised to see shops in Spain close for the afternoon “siesta”. I half-expected this to have changed, given Spain’s economic challenges – GDP growth is slowing down, unemployment rate is the highest among EU nations – but these folks won’t give up their afternoon naps. How refreshing.)

These anecdotes give a different touch to the dominant media narrative about the setbacks or challenges to the green transition in Europe. The challenges are not insignificant (the benefits of more solar power will be limited until the electricity grid is upgraded significantly to store and transmit all that energy, for instance), but the changing attitudes on the ground (and the de-central initiatives being undertaken) allow us to imagine a different future.

So while we subscribe to the data-driven approach and follow journalism that draws more on data, such exposure to the on-the-ground reality seems necessary too. Stats matter, yes, but anecdotes add fascinating and insightful details to this picture of the world we carry in our heads.

I left Europe wondering whether the island of Lanzarote gives us a hint of what life will be like in many places a decade or two from now. Very little water, conditions unsuitable for agriculture… we may have to adapt our lives the way such places today adapt to environment-driven constraints. Keep that Spotify playlist handy.

In July 2021, I moved into a full-time role in the social impact sector after a couple of decades in the enterprise software industry. I’d spoken about this transition – from the corporate to the social sector – with a few friends (one of them over a podcast episode), and I’d read a few articles on the subject. I thought I had a reasonably fair idea of what this would entail. The reality turned out more complex and more nuanced, offering challenges and opportunities I hadn’t expected at the outset. 

The first hints of this gap (in my expectations versus reality) came during my introductory conversations with a cross section of people at Sattva. Given my long tech background, I’d taken some technology knowhow – common in IT circles – for granted, and my speech was sprinkled with jargon I didn’t think twice about. In one such conversation, after listening to me for a while, a colleague asked,

“What’s an API?” 

The question caught me by surprise. It was evident that I had to reset my expectations on tech-awareness and adoption in this sector, and also change the way I presented my worldview in this new space. 

This difference (in worldview, in understanding) existed in both directions. Every third conversation during those initial months brought up the phrase “Theory of Change”: I had not heard of it even once in my long tech career. 

What struck me during those months was the diversity of backgrounds and experiences of most people in Sattva. Unlike the software engineers and product managers I’d worked previously with (who came from an engineering or management background), the picture that emerged from Sattva employees covered a broader palette, spanning social sciences, arts, humanities, economics and a few from engineering and management backgrounds too. And unlike typical IT folks who are driven by market forces – or a mix of tradition and societal expectations – to choose their line of study and work, my new colleagues had consciously chosen their field of work, often out of a desire to make a difference to society.

This diversity in my new workplace seemed apt, given the multiplicity of domains and stakeholders I found among the projects at Sattva. We worked with Foundations, Corporates, Non-Profits, Social Enterprises, Government agencies (among others) across a range of sectors including health, agriculture, livelihood, and education. This was clear from the start, and I understood it intellectually. But a year into my new role I experienced this diversity first-hand during two visits in the same month. 

The first was a field visit to Rajasthan, to observe some education camps set up in remote villages. The project was part of an NGO’s efforts to improve adolescent girls’ education in marginalised communities, and these village “camps” were modest setups: we sat on the floor of a small room with a dozen students, and watched the tutor facilitate a class. Later there was a focussed-group discussion with a set of tutors from that area, again in a similar setting: on the floor of a room with a leaky roof, adorned with photos of ancestors and some framed certificates. A few weeks later, on a different project, I found myself sitting in the coffee shop of a five-star hotel in New Delhi, talking about a proposal with an international foundation which had funded a part of this project. There were expensive works of art around us, and the seating was plush. The contrast between the two meetings was stark, and it showed two different sides of this ecosystem: where the money was flowing from and into. Sattva had to work with both sides, and embrace this contrast. This was integral to our mission. 

Technology was another area where I encountered several differences. In my earlier avatar I had worked for SAP, a software company that built products for medium to large-size enterprises. I’d written about the challenges in building B2B products, but developing and deploying technology for the development sector (Tech4Good, as some call it) revealed facets I’d not seen before. 

There were new constraints to consider, like access to digital infrastructure and the digital literacy of target users.  Further, we could no longer assume English proficiency, even for early adopters. Designing software under these constraints was a new experience, something that made the traditional B2B or B2C scenarios seem a lot easier in comparison.

Availability of talent to build software at a reasonable cost was another challenge. With the tech industry luring people with large pay packages and 30% or higher increases each year, hiring and retaining experienced tech professionals in the development sector requires an approach that goes beyond money as an incentive. (Interestingly, I found that Sattva had some great tech talent – like a computer science BTech from a leading university, who had previously worked on machine learning – but these folks had consciously moved away from tech, and were keen to spend their time and energy in other social sector domains. This initially struck me as a loss, but over time I realised that their background came in handy in projects that intersected with tech. Which, these days, is not infrequent.)

I also discovered a disconnect in the space of what people wanted from technology. There were  conversations where people saw technology as a silver bullet and came up with outlandish expectations, like a WhatsApp add-on that offers real-time translation into several Indian languages in a chat between multiple people. (Even at the current pace of development in natural language processing, this is a few years away). In other instances, I found that users were unable to accept that software could have bugs: when an issue surfaced in an app, the first inclination was to abandon its usage and move to the state before (like using pen and paper, or just a spreadsheet). 

Such mindsets, and the gaps in digital literacy among many users, underscored the importance of change management while targeting adoption of software in such environments. I’ve learned that designing for value, rather than compliance, is a key prerequisite to higher user acceptance and adoption. 

Then there was scale. I’d heard of “Internet scale” before, but in this ecosystem people spoke about “population scale”. The need of the hour, they said, was for solutions that don’t just work well locally but can scale across India. This need was reflected in the significant progress made recently with “Digital Public Goods” or “Digital Public Infrastructure”, like the India-Stack. 

This population-scale thinking is related to a mindset (within Sattva, and this impact ecosystem) of thinking more about the country and less about the company we work for. It is reflected in our projects, like the Bharat Initiative, and products, like India Data Insights. In some contexts, like climate change, the scope and imagination is even broader, encompassing all of our biosphere. Previously, in the enterprise software market industry I was a part of, it was competition that largely drove our thinking (about the company’s strategy and its future); in Sattva, the mantra is collaboration, not competition. 

Transitions of this nature are seldom managed alone. I joined Sattva as a newcomer to the social sector, and my colleagues helped me navigate this unfamiliar territory, pulling me along for field visits, patiently answering my naive questions, and perhaps most importantly, giving me the space to explore and learn. 

All of this makes my earlier occupation seem a lot less exciting, and I don’t think this is an exaggeration. In contrast to the enterprise software sector I was engaged in previously, the impact sector is more diverse, more challenging, not as mature (technology-wise), and offers a large potential for impact in areas where society needs it most. Working in this sector in the last couple of years has given me a level of satisfaction and joy I had not experienced before. I sometimes wish I’d taken the plunge sooner.

(This piece — written in collaboration with Bhushan Nigale — is the fifth in a series that explores the evolution of the software technology stack and software development methodologies in the last two decades. In this instalment Bhushan and I examine the interplay between the old and the new worlds, and also look at how the stack and methodologies play together in this evolution)

The first four articles in this series examined the transition from the traditional stack and Waterfall methodology (common two decades ago) to the New Stack we see today in cloud-native products and the Agile/LEAN methodologies common in software development today. Those articles looked at the drivers (that led to the changes), and the impact (of these changes). Some of the key challenges the new stack and methodologies brought in were also discussed.

Given all this, it’s fair to ask: where does this leave the traditional stack or the Waterfall methodology? Where are they used (or relevant) even today? Do they have a role to play in future? How do these co-exist with the new stack and methodologies? 

This article explores the interplay between old and new, and also how the stack and methodologies play together. 

The traditional stack today

The traditional stack, dominant two decades ago, is still widely in use today. It figures mostly in the enterprise software products built around the 1990s and deployed ‘on-premise’. Some of these products have been rewritten for the cloud, some others have followed the ‘lift and shift’ path to the cloud, but a majority — close to 60 percent [1]— remain where they were originally deployed: in the on-premise data centres maintained by the IT departments of enterprises.  

These legacy enterprise products — and thus the traditional stack they are based on — can be expected to stay operational for decades. The reasons for this are many.

Firstly, the large amount of investment (both in hardware and software) that has gone into these systems results in a lot of inertia. Having invested so much into these systems, the natural inclination is to keep them running for a long time.

Next there’s the tricky matter of switching costs — costs that include not just building or buying new software, but also migrations costs, end-user training costs, etc — that need to be justified: unless there’s a compelling business reason, such transformation projects do not get the budget. 

Then there’s the question of skill. Enterprise IT departments are experienced in maintaining and operating the traditional stack, but they lack skills the new stack demands. Unless there’s a demographic change — which can take decades — this factor will continue to play a role in decisions involving a move to a new architecture.

Ultimately, it is a matter of business priority. These enterprise products also are typically ‘systems-of-record’, which do not face the same kind of demands — to change fast or scale flexibly — as the ‘systems of engagement’ (or, in the B2C world, any consumer facing apps) do. And while they may be mission critical, these transactional systems are often not seen as strategic: so why touch them if most of the innovation is anyway happening elsewhere? As long as the data from these systems of record can be accessed quickly and used (for AI related capabilities, for instance), there’s little business need to rebuild these solutions on the new stack.  

So these legacy products built on the traditional stack will continue to be in use in the foreseeable future. One important consequence of this is the rise of Robotic Process Automation (RPA) tools in the software industry [2]. These tools make up for the deficiencies in legacy software (like missing APIs, or  fragmented toolsets) and add a layer that further removes the need to modernize legacy solution landscapes among enterprise customers. 

Co-existence of the two stacks

The traditional stack and the new one come together in the context of hybrid landscapes. As the name suggests, such a constellation spans products based on different architectures (the traditional and new stacks) and deployment models (on-premise and cloud). In B2B enterprises, such hybrid landscapes are all too common. Legacy products based on the traditional stack come together — in various business scenarios — with newer solutions built using cloud-native architectures.  

Such legacy products based on the traditional stack are also being deployed — using Virtual Machines or containers — in the cloud. This ‘Lift and Shift’ approach involves moving a legacy application (running on-premise) to a public cloud provider like AWS or Azure with minimal cost and minimal to no code changes. The advantages of such a move to the cloud  include reduced costs, higher availability, better agility and improved reliability. It also can pave the way for the next step of being re-written natively for the cloud.

Modernising the traditional stack (or parts of it)

The traditional stack, as we saw in the first article, was superseded by innovations that led to the new stack. But the software community has continued work on parts of that old stack, and today we have flavours of some “old” technologies and methods that are well adapted to today’s environment. 

Perhaps the most visible example here is the relational database. A couple of decades ago relational databases, having survived the hype wave of object databases, were dominant. But they were hard to scale, which was a key requirement of web-based software catering to a large (and growing) number of users. The SQL-based relational database was designed to run best on a single machine, not a cluster of nodes. This gap paved the way for a new class of databases (later placed under the umbrella of ‘NoSQL databases’) that don’t use the relational model and are designed to run on large clusters. But in the last few years, we’ve seen the rise of relational databases designed to work in a distributed cluster. Prominent examples here include Google Cloud Spanner and CockroachDB, solutions which allow one logical database to be distributed across multiple nodes in a cluster. The advantage of distributed SQL databases like these (over the NoSQL ones) is that they are built ground up to offer transactional consistency. So you have all the advantages of the relational database plus the scaling benefits a distributed architecture brings. 

The second example is not a technology per se, but more of a construct. The monolith, thought to be neatly buried under the hype of microservices, has been resurrected following well-documented failures that showed the drawbacks of using microservices in contexts where monoliths were more appropriate.  Even in contexts where microservices eventually turn out to be the better approach, we’ve learned that it’s prudent to start with the monolith and later break it into microservices. The “modernisation”, in this case, is ideological: the old myth about monoliths was gradually replaced with a new one. 

Beyond these public examples, there have been other proprietary (and thus largely unheard of) efforts to modernise the traditional stack. At SAP, where I worked previously, there were efforts made to optimise the ABAP application server — which has been around for decades — to run better in VMs and containers in the cloud. What was earlier a standalone application server is now a ‘multi-cloud’ PaaS; the ABAP runtime is now offered as a deployment option within the SAP Business Technology platform (alongside other runtimes like Cloud Foundry, Kubernetes, Serverless, etc). 

Moving towards a hybrid methodology

As it is to be expected in a widely adopted methodology originally conceived almost two decades ago, flaws have crept in Agile. The agile rituals (daily scrum and backlogs) are the easiest aspects of Agile to ‘adopt’, without having to do the heavy-lifting of fundamental principles and essence of agile, i.e. to put people before processes. Without an appreciation of the essence, the rituals themselves peter away, or are even counter-productive: a Dark Scrum, where older hierarchies continue instead of the self-organization around a mission expected from Agile.

The ballooning of the agile consultancy industry (memorably termed as the Agile Industrial complex by Martin Fowler) has led to a variety of flavors being propagated, with elaborate process steps and as mentioned in article four, a zoo of tools that enforce their own processes. The ‘processification’ of Agile belies its founding principles outlined in the agile manifesto. Lastly, if imposed by management without an explanation (and  the consequent buy-in), the processes might dampen the sheer individual brilliance of gifted developers. 

Partly for these reasons, all its primacy in ‘born-digital’ companies, Agile hasn’t made headways in traditional software development/maintenance organizations. As this report shows, ~51% of surveyed organizations still used Waterfall (always or sometimes).  Some organizations decide to continue to follow the Waterfall methodology, as in certain conditions it makes sense to use Waterfall. 

For instance, the project might be running on fixed costs or timelines, making the uncertainty of delivery dates undesirable. Not all organizations/departments have the appetite  to undergo expensive, long-running agile training (an effect of the agile consulting industry mentioned above) and thus might not have the skills (also owing to a generational gap) to work with Agile. Lastly, for massive, multi-million budget programs running across several departments, management might insist on a structured approach with defined milestones, oversight with regular reports, and steering ability.

In such instances, a ‘Hybrid’ methodology comes to the fore. Via a structured process, sometimes involving external agencies, a project team of specialists collects requirements upfront. Typically, this is a small team of cross functional experts led by an experienced program manager. With an iterative process, the team then outlines the target state of the program – a transition to a new system, a new suite of products, upgrade of a complex system to a new release. After an approval from a steering committee – again, drawn across leadership of departments – teams are set up. 

The teams themselves work in fully agile mode, with a well-defined release backlog, regular stakeholder interaction, and release in smaller iterations. Across the program, coordination is usually done by the program office, with regular updates to the steering committee. As the release/upgrade/cut-over approaches, detailed system acceptance tests are conducted, with a centralized release decision approach.

This federated approach grants autonomy to teams to work in an Agile mode, while still keeping the elements of centralized control over the timelines (usually fixed well in advanced) and implementing across-the-board conditions, such as adherence to guidelines (e.g. use of a particular technology as part of the company strategy).

Conclusion

What we’ve outlined above shows that ample conditions exist for the continuation of the old technology stack and older practices. Situations unique to an organization further complicate the choice of making a transition to a new technology or practice, and overcoming the status-quo sometimes needs actions that have less to do with the merits of a technology or methodology than the operating constraints decision makers find themselves battling with. Decision-makers need to evaluate their position in the matrix below and weigh the risks of making a transition to the new world.

There is a transition underway from the traditional towards the new or a hybrid model, and the paths taken depend on how these organisations navigate the cost-benefit-risks aspects outlined above. This trend is reflected in the options cloud infrastructure vendors (like AWS, Azure or Google Cloud Platform) offer in the space of modernisation of the legacy stack, and the consulting solutions IT service providers offer in this space. 

This shift to the new stack and methodology is a generational change, and technological merits alone cannot drive such a change. 

Footnotes/References:

[1] A 2020 survey from the Uptime Institute showed that 56% of the enterprises run “most of their workloads in corporate data centres — that is enterprise-owned, on-premise facilities.”

[2] According to a Gartner study from 2020, the $1.5 Billion market for RPA solutions is expected to grow with double digit rates until 2024. 

In article two of the series I presented the various forces that have led to the evolution of software development practices from the Waterfall model to Lean and Agile. We saw a variety of proximate causes that caused this evolution: the increasing role software plays in all spheres of our life, the massive changes in software architecture and the mainstreaming of Open Source software, the increasing consumerization of IT and the changing demographics of the software industry. 

This article examines the consequences of these changes. Mainly, it answers the question: did Agile and Lean hold on to their promise? When a species evolves to adapt to its new environment, manifest changes appear. Can we discern such changes in the industry, for instances in workplaces and the roles played by practitioners? If we live in a post-Waterfall world, what are the obvious signposts that the changes have ushered?

In what follows, I provide an overview of how other industries have begun to adopt Agile, to what extent the hierarchies still matter, the roles of teams over individuals, and the rising importance of roles such as the Product Manager.

Agile delivers

The agile movement that arose from the Agile Manifesto is now widespread to the extent that software development organizations consider it the de facto style for delivering innovation at scale. Software development and implementation projects are risky, failure-plagued endeavors: while statistics widely differ, reliable studies (such as the Standish Group’s Annual CHAOS report) report as high as two-thirds of technology projects ending in partial or total failure. 

With its emphasis on involving end-users as early as possible and then collaborating with them, smaller release cycles and a clear articulation of user requirements, Agile addresses the most crucial reasons for these failures: users become stakeholders, vested in the success of the project, rather than just using the project ‘thrown over the fence’ to them (e.g. by IT departments). The transparency in progress improves trust and the health of interdepartmental relationships – truth is the best disinfectant.

Hierarchies matter less 

A counterintuitive, but welcome, change has been the gradual flattening of organizational hierarchies. While Lean originated in manufacturing companies, traditionally hierarchical with a ‘command-and-control’ operational model, its fundamental principle of putting customer value first meant that employees need to be more empowered to ensure this principle lives in practice. Thus, a product owner several levels below the unit head, takes significant decisions and takes accountability in the success of the product: brand new announcements in products and cloud services are increasingly made by Product Managers and not development departmental heads.

Managers themselves stepped back from their hands-on, roll-up-the-sleeves style to a ‘coach’ role, imparting wisdom gleaned from experience and focussing on developing team members. Not all managers have managed this transition and the apportioning of their responsibilities successfully. 

Adoption of Agile widens

This success of Agile – despite the inevitable failures that we address later – has led to its increasing acceptance and adoption in disciplines outside of software product development. While the results have been decidedly mixed, traditional functions (as opposed to newer product lines/new businesses with focus on innovation) are increasingly adopting Agile principles, as evinced by the widespread use of the terms ‘scrum’, ‘backlogs’ and ‘daily stand-ups’. Large IT departments are moving away from a purely project-based implementation mode to a hybrid mode, with a combination of Agile and traditional project management approaches. 

A similar transformation has occurred in companies embracing Agile, as pressure has increased on traditional companies – the ‘not-digital-born’ companies – to deliver faster and in quality. Departments outside of IT enthusiastically begin to embrace Agile, as the recent story on Mozilla shows – reportedly, the Open Source company runs its marketing department using an approach inspired by Agile. 

Learning and Development departments too have begun to include Lean and Agile in their approach, with Agile learning being promoted as an “approach to training and development that focuses on speed, flexibility and collaboration.” Even staid, process-intensive sectors such as Pharmaceuticals, are gradually embracing Lean as expectations mount to deliver pandemic-combating vaccines while reducing inventory levels quickly. 

Winds of change are even blowing in NASA. NASA has its own Waterfall variant, with detailed policies and guidelines as the system progresses through the development life cycle. However, for the planned 2024 moon landing program, NASA uses Agile. The Orion Multi-Purpose Crew Vehicle team, responsible for checking critical functions of the spacecraft, is fashioned using the ‘self-organizing teams’ principle. The project lead reported faster decision making, cost savings and flexibility, resulting in a better product.

A fusion of Flows 

Another central tenet of Lean found a wider resonance: Flow. Lean Flow stipulates how items in a process move from the first step to the last, as quickly as possible and in quality, to ensure customer satisfaction. With the direct and early involvement of users, teams began to firsthand observe and experience impediments to Flow: idling, unfinished work that assumed forms such as a highly detailed requirements document, in-depth design and architecture discussions, a long-drawn testing plan, etc. This introspection – triggered by the Retrospective session fundamental to Agile – led to a rethinking of the software delivery process to eliminate waste, both obvious and camouflaged. (The NASA team mentioned above discontinued sprints later, but continued their retrospective sessions, as it aided faster decision making.)

Thus, when a team truly delivers value in a  smooth iteration, it experiences another type of Flow, first described by Mihály Csíkszentmihályi (albeit in another context): a team delivering value is so fully immersed in a feeling of energized focus, full involvement, working in full harmony with each other, and enjoyment in the process of the activity that it loses track of time.

Flowing to DevOps 

When applied to the delivery of cloud-native software services, the Flow principle expands the horizon of a team/organization’s responsibility to ensure the successful operation of software: the value is realized only when the software is available. Agile has thus paved the way for DevOps, the movement that emphasizes tighter communication and cooperation between software developers and other information-technology (IT) professionals, most notably Operations.

Viewed with the lens of the Flow principle, teams not only create and deliver cloud software, but also ensure its smooth operation, and shorten the development cycle and improve the quality, using techniques such as Continuous Integration and Continuous Delivery. The success of largely autonomous, cross-disciplinary teams has paved the way for what was unthinkable when I began my career in the late 1990s: a unified view of development and operations (“DevOps”) rather than silos. Many operational tasks are unplanned and critical (“live site first”), and the Agile approach (one backlog, customer value first, extensive communication) provides similar benefits for these unplanned tasks as well as it does for planned tasks: Agile and DevOps work beautifully in tandem. DevOps aims at establishing a culture where the interdepartmental boundaries/silos between development and Operations are torn down.The resultant convergence of people, processes and an impressive variety of practices have promoted agility. 

Teams over individuals

Software skills are increasingly specialized, but complex problems demand that specialists in various domains work together, especially if the solutions are unclear and requirements are fast-changing. This has led to the rise of multidisciplinary teams that are largely self-governing, as opposed to a ‘project task force’ or even a ‘rockstar developer’. This in turn has led to a partial-to-complete democratization of the software development process: less-to-no code ownership, an ability to challenge unrealistic timelines and a gradual shift of decision- making to teams. Teams even feel emboldened not to ship a release not meeting the ‘done criteria’.

A happy side-effect has been (at least until the time people worked in offices) the more communal feeling Agile lends to workplaces. Even the youngest/newest team member has a voice – the ritual of the daily stand-up ensures this. Colorful Kanban boards, the sometimes-festive celebration of results with a sprint review not only bring transparency, they create a sense of forward movement, of togetherness. (It is hard to feel enthusiastic in the same way about a project steering committee meeting.)

Thus, when the normal working mode was upended by the COVID-19 pandemic, development teams could easily switch to the changed realities, and (anecdotally) did not experience sharp drops in productivity as they already had considerable experience in self-organizing themselves. Impressive tools in the arsenal helped here: the practices of having a well-defined, prioritized list of items via the ranked backlog, visibility in the topics via a Kanban/Scrum board, regular updates via a daily standup, and various roles where the responsibility was distributed (as opposed to a hierarchical structure). 

In summary: agile practices promote team effectiveness over individual brilliance. Complexity in enterprise and consumer software has exploded. No bunch of individuals, however gifted they might be, can steer the ship, and development teams are increasingly cross-functional rather than departmental. Members of self-organizing teams iterate over a plan-do-check-act cycle using effective communication means such as sprint planning, daily scrum, sprint reviews and retrospectives. 

Tools ecosystem  

The software industry is unique in its relationship with tools. We are fully vertically integrated: not only do we have software and tools for building and testing software (IDE, compilers, etc.), distribute software (online marketplaces), market software (websites, social media)  but also tools for how to build software. A case in point has been the use of Kanban (Japanese term for a signboard). People have used post-it/sticky notes to remind themselves of important tasks. The Kanban system took the idea further for a team. Thus, using sticky notes on a whiteboard, teams create a Kanban system, bringing in visibility on the topics at hand and their priority. Team members can then pick up tasks, lessen duplicate work, and increase the agility, thereby achieving Flow. 

Having sensed the increasing demand for Kanban, DevOps and related Lean-based tools for organizing Flow, both open source and commercial software vendors have moved quickly to offer a litany of tools. Once limited to version control systems at best, ‘software to build and operate software’ has now expanded to cover a range of activities, from backlog definition management (e.g. Jira) to Kanban (e.g. MS Planner), to CI/CD pipeline management (e.g. Jenkins) to Application Performance Management, culminating into DevOps toolchains. 

Thus, a software practitioner – irrespective of their function – needs more than a passing familiarity with the tools, and based on the role can spend up to hours per day navigating the tools and the processes around them – an irony not lost on more seasoned practitioners. The temptation to throw a tool at a problem is often difficult to overcome, as scores of stale Jira items and ‘Planner’ Kanban boards testify.

Role of the Product Manager 

Of all roles that arose from Agile, arguably the one that gained most prominence is the Product Manager. Product Managers play a pivotal role in the end-to-end lifecycle of digital products: they conceive, define, test, deliver, and measure (and occasionally recall) products in the market to maximize business results. They are expected to analyze the market, hunt for demand signals and distill them into precise requirements, enlisting practices such as customer research. Then they work with key stakeholders across business and technology to further their product vision. With all ‘go’s in place, they translate this vision into the broad themes and epics that the teams would develop, with the help of product owners. Together with product teams they deliver and grow the end-to-end customer experience, closely safeguarding customer value. 

Their remit is therefore broader than the product owners. Product Managers sometimes have a business background, but as the technology stack articles in the series have shown, technical knowledge is increasingly a prerequisite for success. Above all, product managers quickly learn that cloud-native software operates on circularity principles as opposed to the traditional linear models of value delivery. Users generate a significant amount of ‘digital ballast’ while using digital products. Using telemetry, logs, feedback, etc., product managers have a much stronger insight into usage patterns in their software as opposed to on-premise software, and can use this data to improve their product.

Conclusion 

In this article, we saw the major consequences of Agile and Lean. A common theme underlying these laudable consequences has been the emergence of a new culture: adaptive, responsive and flexible with demonstrably increased trust and transparency.Though all ideals of such a culture have not always been met in practice, the cultural transformation towards agility “putting customer value first”  has been visible. However, as Agile becomes entrenched, it confronts an inevitable challenge – Agile is being touted as the perfect solution for all software projects, irrespective of size, flavor and operating conditions: in short, a ‘silver bullet’. How realistic are these expectations? Why doesn’t every Agile project succeed? Do Waterfall elements still have a role, and what are the conditions in which Agile can interplay with Waterfall? We turn to these questions in the following article of the series.

The first article in this series began with an outline of the “traditional” technology stack that was common in the early 2000s. It then examined how the internet, mobile, and cloud revolutions exposed the limitations of this stack, deficiencies that led to the new stack we see today. The article outlined the key characteristics of the new stack, and we also saw how these traits solved problems this traditional stack could not.

The stack today looks very different from the one we saw two decades ago. It consists of small, loosely-coupled (and mostly open-source) pieces that are distributed over a network and communicate using APIs. These aspects — the breakdown of the stack into smaller components, the ubiquity of APIs, the widespread adoption of open-source, and the distributed architecture — have had a huge impact in the last decade or so. This article will look at these consequences, both positive and negative.

First-order effects

Perhaps the most important consequences (of this breakdown of the traditional stack to the new one) have been the creation of a software supply chain and an API economy.

With the traditional stack, it was common for vendors to build most parts of the stack themselves. Vertical integration was seen as a competitive advantage, and software companies like Oracle even acquired hardware vendors (like Sun Microsystems) to offer the full stack, from infrastructure to user interface. And it was common for enterprise consumers to go to a small set of vendors to meet their software needs.

What we see today — thanks to the new stack that leans towards single-purpose solutions — is a best-of-breed approach for constructing the stack. Vendors (or open-source projects) offer specialised solutions or frameworks across the stack and across different stages of the software lifecycle [1]. The entire supply chain of software — from planning, development, delivery, to operations — can now be composed of tools from niche vendors or open-source offerings [2]. This trend highlights the growing maturity of the software industry: we’ve gone from a model where most parts of the solution come from one vendor (or a few vendors) to a model where a rich ecosystem of vendors is powering the entire software supply chain.

This explosion of choice (and the explosion of start-ups offering this choice) is one feature of the API economy. Several capabilities have come together to make this possible: standardised communication protocols, easier ways to publish and consume APIs, tools to manage the API lifecycle, among others. APIs are now the main building blocks of solutions built on the new stack.   

In parallel to this broadening of the tools ecosystem, we’ve seen the rise of software platforms. By offering ‘opinionated’ ways to build solutions, platforms pave the way out of this conundrum of choice. Platforms also offer a wide array of integrations that can grow over time, keeping pace with the growing number of vendors offering tools in each domain. As one provider put it, “a platform creates a stable center of gravity for your tech stack.”

The widespread availability of APIs has raised the level of abstractions used to build software. The software primitives we use today make it a lot cheaper and faster to build and manage complex software solutions. The emergence of low-code and no-code tools is another manifestation of the higher level of abstractions available today via APIs. Entry-barriers for developers are now significantly lower than what it used to be with the traditional stack. And a new class of software creators has emerged, driving the so-called ‘citizen-developer’ movement. 

This API revolution has also led to more software using software.  Driven by the need for more automation, software has taken over large parts of activities done previously by humans. These days it is common to see software tools setting up server farms (via infrastructure-as-code solutions like Terraform), bots automating complex workflows (via Robotic Process Automation solutions like uiPath), and even bots orchestrating other bots [3]. The new stack, in contrast to the traditional one, has a far lower need for humans doing such orchestration. 

The API economy has enabled the growth of marketplaces where these APIs (and their underlying software components) are published and consumed. Efficiency gains driven by the new stack (like zero marginal cost of adding a new user) also enables consumers to try out these components before they buy it. Such software marketplaces — and their adoption models — were unthinkable two decades ago, given the architecture of the traditional stack. 

Second-order effects

The new stack has had some second order effects that are worth examining. 

Compared to the situation a couple of decades ago, today we see the demand for more specialised skills across the stack. Apart from the obvious specialisation in front-ends and back-ends, the importance of understanding how to build and operate distributed systems has grown significantly; what was once a mostly academic pursuit is now at the heart of the software architect’s practice. On the flip side, all this specialisation has also increased the value of people who can work on the entire stack. ‘Full-Stack Developer’ was a meaningless phrase two decades ago, but these days it’s a skill that carries weight. 

Site Reliability Engineers (SREs) and DevOps specialists are in high demand to manage the operational complexity of the new stack. These disciplines, non-existent two decades ago, are the result of new practices that have evolved in the last decade or so — a topic that will be covered in a separate article on engineering practices. 

In Product Management, the new generation of internet/cloud/mobile applications has led to a shift towards relying on data and quantitative methods of evaluating impact. Measuring usage/adoption, defining metrics and tracking them are practices that are commonplace now. 

Another impact has been in the new types of applications that the new stack has enabled. 

As the volume of data gathered increased over the last two decades, we saw the rise of machine learning techniques that took advantage of Big Data. Without the distributed architecture of the new stack (and the economies of scale it led to), it’s hard to imagine machine learning reaching the maturity it has today. BlockChain is another application type enabled by the distributed nature of today’s technology stack. 

It’s easy to miss the steps that led to these technologies, the factors that made them effective. While the drivers that led to the new stack tell one story (of how certain kinds of applications drove the change in architecture), the consequences tell another one: of how the new architecture in turn led to new types of applications. Machine learning and Blockchain were not in the minds of engineers who adopted a distributed approach for internet based applications, but that approach is what made these new technologies possible.  Distributed storage techniques and the compute power we have today in the cloud are key enablers for both technologies. 

Another direction the new stack has led to is the rise of ubiquitous computing.  By unshackling itself from its ‘on-premise’ roots, the new stack paved the way for the stack — in its different forms — being present ‘everywhere’. The spread that began with mobile has now reached every imaginable physical device that needs to be controlled. 

Challenges

The new stack solved many of the problems the traditional one could not, but it also created new ones. Most of these were the result of the split of the stack into small pieces across the network: in other words, the challenges of building and managing distributed systems. 

Integration, which was a challenge even with the traditional stack, grew more complex with the new stack. Business processes that were earlier running within one application are now spread across multiple services in the network. These services can come from multiple vendors, and run in different “clouds”. In such a setup, the first challenge is to wire together these independent services into a coherent business process [4]. Next, getting those business processes to reach the same level of consistency as before is a lot harder; transactional data, now spread across multiple databases, brings up new challenges in data consistency and aggregation. Managing master data — also spread across different services in the network — becomes harder too. 

Operations is another major challenge with the new stack. Distributed systems are not easy to understand, observe and manage — there are many points of failure, and the monitoring data generated comes from heterogeneous sources, which makes it hard to find the root cause of an issue. Microservices complicates this picture further: what was earlier one (technical) process in a monolithic application is now probably a dozen processes that need to be observed and managed. And the increase in the number of abstractions in the stack has also driven up complexity: from an application running inside a virtual machine, we have evolved to a bunch of services running in containers that are managed by a container orchestration tool like Kubernetes and also orchestrated via a service mesh like Istio. The growth of the ‘observability’ tool ecosystem — with dozens and dozens of tools to monitor cloud-native environments [5]  — shows how much more complex (and important) this space has become. Things were a lot simpler with the traditional stack. 

Security (and in particular Identity and Access Management) is another key challenge the stack brings up. Earlier, with applications on the traditional stack, there were a small number of resources (applications, data), running on few servers, and used by a (relatively) small number of users. This has now ballooned into a complex matrix spanning a large number of resources, distributed over a cluster of servers, and used by a vast number of users (at internet scale). The resulting whitespace has led to the rise of firms like Okta and SailPoint who have taken up the identity and security challenge in the cloud.   

Conclusion

This tour reveals that the comparison between the traditional stack and the new one isn’t really black and white. While the new stack ushered in a lot of benefits and changed the IT industry landscape, it has also brought with it several challenges that we are still grappling with. 

No wonder, then, that the traditional stack — or many elements of it — are still around, and are not going away anytime soon. In fact, some elements of the traditional stack — like the relational database — are making a comeback in this new world of distributed systems. Will the traditional stack be able to revive itself? In what contexts is it still relevant? 

In article five of this series, we will take a closer look at these aspects and examine the interplay between the traditional and new stack. 

Footnotes

[1] Examples across the stack include Docker and Kubernetes in the infrastructure layer; MongoDB, PostgreSQL, Redis on the database layer; Spring Boot at the application layer; AngularJS or React at the UI layer. A zoo of tools has emerged also in the space of managing the lifecycle of cloud-native applications: Terraform for infrastructure setup, Jira for issue tracking, GitHub for versioning, the ELK stack for observability, StatusPage for incident communication, and so on. 

[2] It is striking to note that even a capability like publishing release notes is now a small domain in itself, with multiple start-ups attracting VC funding. Examples include AnnounceKit, ReleaseNotes, Noticeable.

[3] This level of automation is giving rise to segments like B2R2C — Business to Robot to Consumer — where bots make buying decisions for consumers. Now marketing folks need to target bots, not just humans! 

[4] Tools like Zapier, IFTTT, Tonkean, etc. allow the wiring of a host of other tools into a complex business workflow. 

[5] Examples include Dynatrace, New Relic, Data Dog, Splunk, Prometheus, Honeycomb, the ELK stack — the list can go on. 

A benefit of spending over two decades in an industry is that one develops a perspective to separate hype from substance. This viewpoint is especially useful in an industry like software, where minor feature increments are hailed as innovation, and press releases, blogs and Tweets tout routine upgrades as revolutionary. After having lived through several such hype cycles that have a high probability of going bust, one learns to exercise caution, and appreciate genuine path-breaking innovations (the first article in this series — written by Manohar Sreekanth — lists the technology changes that have stayed).

Innovation in software development methodologies is even harder to achieve and sustain. A paradigm shift is rare – at least in the original sense of the term (Thomas Kuhn used it to define  a fundamental change in basic experimental practices in a scientific discipline). Inertia is difficult to overcome, especially if established methodologies seem to be getting the job done.

I’ve been privileged to witness and experience firsthand such a paradigm shift in software development, namely the shift from Waterfall to Agile methodology. The shift has been so complete that new entrants to the industry have little – if at all – any familiarity with the older methodologies. Agile is their new default mode now.

Examining and reviewing this shift is both useful and important, because the promises of any established order need to be constantly reexamined as flaws and digressions inevitably creep in. Over time, unless tended carefully, practices tend to return to older routines — regression towards the mean is an iron-clad statistical law. Understanding the older practices and the change drivers that led to their evolution help us better appreciate the advances and detect costly deviations. An appreciation of the historical developments helps practitioners not only to address flaws, but also iterate over the methodology to adapt to the changing operational environments.

A variety of forces have led to this evolution from Waterfall to Agile: the increasing role software plays in all spheres of our life, the massive changes in software architecture and the mainstreaming of Open Source software, the increasing consumerization of IT and the changing demographics of the software industry. We examine these factors in this article, and treat the consequences of these changes in a subsequent one. 

From Waterfall to Agile

The previous article in this series traced the fundamental change in the technology stack used to build software applications. A parallel evolution, in the methodology of developing software, has accompanied these mammoth technological shifts.

When I entered the industry in the late 1990s, Waterfall had none of the negative labels one finds associated with it today. Terms such as ‘Software Requirements Document’ and ‘Handover to Maintenance’ were ubiquitous and carried a certain respect – passing a Quality Assurance Gate was a big milestone that invited celebration. The software development process flowed from a high perch (hence ‘Waterfall’) of analysis and design to the plains of testing and release, where software was then finally delivered to customers.

But cracks had already started to appear. Disenchantment was rising, both with long delivery cycles and the obsession with adherence to the strict development processes. The internet – which broke the traditional stack as we saw in the previous article – was triggering foundational changes in which software was consumed, and these consumption-driven pressures were now being transmitted to how software was being built. Consumers wanted their software delivered to them faster and better, even as it began to occupy an increasingly central part in their lives. 

Anxious to bring more predictability, agility and quality in the delivery of software, practitioners started to look at other industries that had an enviable track record of delivering value to customers in a rapid, sustainable way. In particular, the automotive industry’s production processes, inspired by a cluster of methodologies first used and honed at Toyota, stood out. LEAN thinking put customer value at its heart, and a cultural transformation – a mindset shift at its center. Various authors, such as Tom and Mary Poppendick, helped translate LEAN thinking and its core principles to the software development context.

Thus, when LEAN came in the aughts to the large enterprise software company where I worked, the excitement in development units was palpable. LEAN’s promise was simple: focus on working software, rather than working processes — customer value over process compliance. Its core principles were easy to understand — ‘flow’, waste elimination, smaller batches, ‘pull’, respect for people and culture, and relentless improvement. Putting this fundamental paradigm shift in practice proved far more difficult.

Fortunately, by then the Agile movement was gaining credibility. The Agile manifesto succinctly codified the principles that defined a new approach towards iterative software development and began to bring the benefits of these innovative methods to the industry. (Prominent software practitioners and leading thinkers in the software industry had drafted the manifesto in 2001 – a watershed moment in the history of software development.) In contrast to the Waterfall, Agile practices are iterative instead of rigid: what they seemingly sacrifice in the security of strict adherence to plans they more than make up in delivering customer value at scale and speed.

This combination of LEAN and Agile and its application to the software development process fundamentally changed the way we would build software. These methodology changes have been profound, and as I argue later, have helped software practitioners deliver immense value in these two decades. Similar to the evolution of the technology stack, these methodology changes have been large enough to discern a break between traditional and new aspects, as tabulated below along the phases of a product life cycle.

With the benefit of hindsight, we now have a clearer appreciation of the variety of changes that continues to drive this shift: the ubiquity of software in almost every sphere of society, which in turn has led to a changing nature and expectations users have from software, the profound changes in software architecture and the accompanying changes in the consumption of software, and finally, the changing profile of software practitioners itself. In the following sections we explore each of these drivers.  

Primacy of Software

 A significant portion of humanity now spends a substantial portion of its daily time interacting with software, often without knowing it. Governments and enterprises were already at the forefront in exploiting software to reap benefits of efficiency, speed and productivity. Advances in hardware and software have not only accelerated this trend in enterprises, it has long permeated in our vanishingly small private spheres. We find software embedded in what were once purely mechanical processes enacted over physical means – driving a car, reading a newspaper, cooking a meal. ‘Cliff events’ such as COVID-19 have further accelerated the digital transformation of our lives, as the physical world cedes to the virtual.

With software becoming central in large parts of our professional and personal lives, this shift in agency to software has led to profound implications in what we expect from software. We are all consumers of software now. As consumers we expect a faster time to value, high satisfaction and place a premium on non-functional requirements such as availability, performance and security – even though we might not be expressing our expectations in exactly these terms.

Traditional development practices such as Waterfall simply could not cope with this evolutionary spiral. Much like software, Waterfall had its roots in government and defense, where stakes were considerably higher, with multi-million-dollar budgets and hundreds of specialized programmers. A scientific, rigid process smoothened information flow and brought predictability and reliability (at least, that was the idea). NASA adopted Waterfall with vigor, and its success blazed a trail: if Waterfall landed a human on the moon, it was good enough to manage inventories and invoices.

But when these inventory status checks and invoices started to appear on mobile devices, consumers started to demand the same level of usability for enterprise software as they were accustomed to in their private lives. Pleasing, functional design, speed and performance, wider functional range (“there’s an app for that”) became norms. Users also began to expect regular updates to their apps – without the long upgrade cycles. Decision makers began to demand the provisioning of functionality hitherto available only via desktop applications on their mobile phones: if they could respond to urgent requests via EMail on their device, why couldn’t they approve purchase requisitions and investment cases?

The parallel shift happening to on-Demand deployment of software meant that release cycles that were spread over months (if not years) suddenly could be compressed to weeks, days and even hours. Waterfall, with its inelastic development and release phases simply could not keep up with the needed swiftness and the speed of technology changes we’ve seen in the past two decades.(A case in point are the Microsoft Office releases. Office updates were once events, with large IT departments rolling out the upgrades in batches. Now hardly a month passes by without the Suite being updated.)

Increasing consumerization of IT

As users became consumers, buying power for IT also began to shift: Lines of Businesses now play an increasingly important role in buying and deploying software within enterprises, often supplanting the role of IT departments. IT departments reacted in turn sharply to avoid the rise of non-standard IT landscapes (‘shadow IT’), understanding the needs of agility and responsiveness. 

The ‘product manager’ role began to evolve, gradually eclipsing the project manager role in importance. This meant that business outcomes assumed higher importance than successful project execution or illusory progress – product demos gradually began to replace complex status slides filled with ‘traffic light reporting’. A single demo was worth hundreds of slides, and software development teams began to invite internal stakeholders and even customers (based on their relationship) to product demos before releasing/deploying the software. An improvement in developer tooling helped as well, with sophisticated wire-frames and mock-ups helped product managers to communicate product vision and ambition in more grippable/tangible terms. Customers welcomed the ensuing openness and transparency and being able to shape the software to fit their needs; developers enjoyed the early, direct feedback.

Software Architecture Changes

This agility grew lockstep with the accompanying changes in software architecture and deployment that continue to occur. Within a few years of the publication of the Agile Manifesto there emerged new architectural principles that both addressed and fuelled the trend of software increasingly being used by software itself [1]. These styles/principles encouraged the design, development and test of software applications in isolation from their eventual target deployment set-up.  Technology to develop independently and coherently was thus a key enabler for the rise of Agile practices.

This unfolding progression in software architecture found a mirror in software delivery. Plummeting communication and hardware costs meant that enterprise software could also be deployed – analogous to consumer software – on the Cloud and not delivered to the premises of customers. Software vendors managed and maintained software on their own premises and terms, rather than customer IT departments and their partners.  

Taken together, these twin-trends further helped the appeal of Agile. As the vendor manages and operates the software, customers expect to see progress quicker. But this shift of ownership of the deployment phase meant vendors had a bigger freedom in deciding when and how frequently new features could be deployed. Decoupled development between independent modules made possible by APIs, Microservices and the Open Source Software meant that development teams could not only implement their changes in isolation, they could also decide when to deploy these changes to ‘production’. The need to plan with extreme detail large scale release cycles — over hundreds of modules and spanning years — simply disappeared, and the DevOps movement stressed the need to think beyond the immediacies of development and release towards ensuring successful operations of the software.

Open Source becomes mainstream

The parallel rise of open source software also contributed to the adoption of Agile practices. The Open source movement promoted sharing of knowledge and code in a community, with the purpose of developing and improving software. As communities grew, informal roles within the community began to crystallize: product developer, customer, contributor, user and ultimately, the community itself became a product manager.  Tying these informal bonds together wouldn’t simply have been possible with the traditional ‘command and control’ approach commonly seen in Waterfall. 

Professionals moonlighting/contributing to open source witnessed its meteoric adoption and began to wonder whether the openness in knowledge sharing and informal hierarchies couldn’t  be replicated in the enterprise setting. As ‘Open Source took over the World’ with its commercial success and a demonstrably viable business model, its principles (and its products) became more mainstream. Organizations even began their tentative steps towards adopting the open source philosophy internally (“inner source”) with product teams opening up their code to other product teams for modification.

Developers and managers alike have enjoyed the robustness of open source software, tested thoroughly in the community. While the velocity and predictability often come in question with longer, infrequent release cycles, they appreciate the transparency of the backlog – again, hotly pondered and debated in the community, and the improved time-to-market.

Changing demographics

A lesser commented upon factor for this evolution has been the changing profile of the average software developer. The ensuing generational shift from ‘Gen-X’ to millennials, improved (but not-quite-there-yet) number of women choosing software development as a career, the increasing democratization of development tools are primary factors that are responsible for development groups becoming more diverse. Traditional command-and-control structures underpinning Waterfall offer less room for creativity for such diverse groups. For instance, Millennials are understood as being more attracted to organizations that offer an engaging, stimulating work environment. By providing more autonomy to teams and therefore individuals, Agile offers a more outcome-driven work environment to these groups. Moreover, the spectacular success enjoyed by hundreds of born-Agile start-ups like Netflix and Spotify have tempted large organizations to follow suit. (With the “fail fast, fail forward” mantra, even failure seems to have an allure, as long as agility is maintained.)

Summary

The foregoing overview outlined the main factors that led to the evolution and widespread adoption of development methodologies. Not a single factor can be singled out as a proximate cause: it is the interplay of these factors that have ultimately caused the ongoing evolution. For instance, while almost every post-war decade has furthered the role of software in society, and software architecture has constantly evolved, it is only in the past two decades that we’ve witnessed the step changes in consumer taste and the success of internet-based business models. 

The evolution is far from complete, however. These changes in turn have led to a changed environment for software and its practitioners to operate. In the fourth article of this series, I examine some of these consequences and answer the question: have Lean and Agile kept their promise?

[1] Domain Driven Design (2004), Hexagonal Architecture (2005) and Microservices (also 2005). Roy Fielding’s seminal 2000 dissertation on REST had already provided the architectural framework and formalization of interfaces for interoperability of software systems that freed developers from making assumptions about other modules without having to wait for them to function to be built first. 

Bhushan Nigale

One of the privileges of working in the same industry for a couple of decades is that you can look back and reflect upon the changes you’ve seen there. But this isn’t something that comes easily to us. Why are things the way they are in software?  is a question we don’t ponder enough. For youngsters entering the industry, current challenges may seem more relevant to study than past trials. And for veterans who’ve seen it all, the present carries a cloak of inevitability that makes looking at history seem like an academic exercise.

But it doesn’t have to be that way. Understanding the forces that led to the evolution in software we’ve seen in these last two decades can help us make better decisions today. And understanding the consequences of these changes can help us take the long view and shape things going forward. To see how, let’s begin with the technology stack that was common two decades ago.

The Traditional stack

When I started working in the enterprise software industry back in the late nineties, the software we built was deployed on large physical servers that were located ‘on-premise’. The application was a monolith, and it used an SQL-based relational database. The fat-client user-interface ran on PCs or laptops. Most of this stack was built on proprietary software. Put simply, this is how the stack looked like:

This was the state of the client-server computing model used in business applications in the nineties. At SAP, where I worked, the client was based on a proprietary framework called SAPGui; the application server was another proprietary piece of software that enabled thousands of users working in parallel; the database layer was open (you could use options like Microsoft SQL server, Oracle DB, or IBM DB2, among others); and the infrastructure beneath was an expensive server (like IBM AS/400 or Sun SPARC) that sat in the customer’s data center. 

This architecture was optimized for the needs of business applications that evolved in the nineties, and such a stack — from SAP or other vendors in that era — is still used in a majority of on-premise installations. But in the second half of the nineties a different story was unfolding elsewhere. 

Internet-based applications were gaining traction as the dot-com era blossomed, fell dramatically, then picked up again (no longer bearing the ‘dot com’ label). And for those applications, the traditional stack proved woefully inadequate. The reasons included cost, availability, performance, flexibility, reliability, and speed: key demands placed by the new types of applications being built on the internet.

The internet breaks the traditional stack

The internet ushered in a scale that was unimaginable in on-premise enterprise software. Websites like Google, eBay, and Amazon had to serve a large number of concurrent users and cope with wide variations in demand. With the traditional stack, adding more capacity to an existing server soon reached its limits, and adding new servers was both expensive and time-consuming. In the new business context, infrastructure costs could no longer grow linearly with user growth: applications needed an architecture that enabled close to zero marginal cost of adding a new user; the old way of adding expensive hardware was unviable.

The internet also placed a much higher demand on availability: these applications needed to be “always on”. Initially a requirement mainly with B2C applications, availability caught up with the B2B world as consumerization of IT gained speed. Soon ‘continuous availability’ turned into a competitive differentiator for businesses that moved (partially or fully) to the web. Five nines or six nines (99.9999 % availability) became the benchmarks, and a new architecture was needed to achieve this level of availability without driving up costs. Again, the old way of installing expensive servers for failover was simply too expensive and inefficient.  

The need to scale applications better also arose due to performance expectations from internet-based (and later mobile) applications. E-commerce applications also saw peak usage in some periods (like Christmas or Black Friday), and others had ad hoc expectations (like planning an ad campaign for a few weeks). Meeting this unpredictable demand needed a different level of flexibility in resource allocation, something that the traditional stack — and hardware-based methods — could simply not offer.

Businesses that moved to the internet also had to evolve much faster than the systems of record (built on the traditional stack) that had dominated the previous era of business applications. Parts of the application that needed more frequent changes had to be deployed independently — and at a different pace — from other slow-moving parts. This was not possible with the monolithic applications built on the traditional stack: it required a new architecture that allowed teams to build and deploy smaller pieces at a faster pace. (It wasn’t just the technology stack that was inadequate — the traditional waterfall model could also not cope with this pace of change and the flexibility this new world demanded. This parallel evolution of development practices will be discussed in a separate article.)

The internet also spawned companies with new business models. And such companies, based entirely on the internet, demanded a different level of maturity from the technology stack. Resilience in particular was key when the entire business was on the internet, and this needed a fault-tolerant architecture. The traditional stack fared poorly in this matter, as there was little or no redundancy in the setup.  

As internet-based applications grew in popularity, their massive adoption resulted in huge volumes of data that needed to be managed. Here again was a need that the traditional stack was unable to meet, as the architecture — with hard-to-scale relational databases — could not cope with adding more storage space at reasonable costs and with the required agility.

Mobile and Cloud

With the rise of smartphones, applications on mobile devices pushed the above expectations further. For apps running on mobile devices, the stack had to support even better response times. Mobile user experiences also demanded native user interfaces (specific to the device) and an always-on expectation. The smartphone era led to an explosion of B2C applications, which fueled innovations in user interface technologies at a pace that the old (proprietary, fat-client) UI frameworks could not match. 

The growth of the public cloud, driven initially by AWS, accelerated the trends that had begun with the first generation of internet-based applications.

The traditional stack simply could not offer the capabilities such cloud-native and mobile-enabled applications needed. A new architecture evolved with the growth of these demands.

Hardware costs

The emergence of low-cost, commodity hardware (that could be put together to build large distributed systems) tilted the scales further towards this new architecture. Innovations such as Google File System (2003) and Amazon’s DynamoDB (2007) were built on this foundation of cheap hardware, which were just small servers with basic compute and storage. As we will see in the next section, one key aspect of this new architecture was that it was distributed, and these innovations laid the foundation for the architecture we see in the cloud today.   

The New Stack

So what does the new stack — one that can meet the above demands — look like these days? The infrastructure underneath the application is a cluster of distributed nodes that host one or more containers. The application itself is split into microservices that communicate over the network through API calls. All-in-one, relational databases have been superseded by single-purpose NoSQL ones. User interfaces are largely browser-based or natively built for mobile environments. New interfaces like chat and voice are going mainstream. And APIs are first-class citizens by themselves.

Of course, the new stack has not entirely replaced the old one. New applications built today may still have elements of the traditional stack: monolithic applications, for instance, remain relevant in contexts where microservices add unnecessary overheads. But today if one wants to build a cloud-native application that must scale to a large number of users in a cost-efficient, flexible, and resilient manner, it will contain most of the new stack elements.

Key characteristics of the new stack

While the above stack picture outlines the individual elements of the new stack, there are some common themes that run across these layers. Let’s look at some of these.

The new stack is distributed. The infrastructure layer is a cluster of smaller servers made of commodity hardware. The application layer is a collection of microservices running in that cluster, each of which may consume services from anywhere in the cloud. The database – and the data – is spread across the cluster. These elements make the new stack more loosely coupled.

The new stack is all about APIs. Each layer in the stack is a collection of APIs that define the behaviour of that layer.  The new stack is also ‘software-defined’: everything is code, including the infrastructure. 

The breakdown of the traditional stack into smaller pieces has led to focused solutions that solve a more narrowly defined problem with better efficiency. NoSQL databases (like graph db, document db, etc) are good examples of such specialisation, but it can also be seen in the UI layer with specific (native) user-interfaces that offer a better user experience than a general-purpose client. In other words, the new stack offers specialised/single-purpose solutions over all-in-one/general-purpose solutions we saw earlier.

The new stack relies heavily on open-source software. The breakdown of the stack into smaller pieces was in part the result of and in part fueled by the explosion of open-source projects that offered smaller, reusable parts and specialised solutions (like MongoDB for document storage, Infrastructure-as-Code offerings like Terraform, etc). In fact, the broad adoption of the technology stack we see today owes a lot to open source. It offered a more pragmatic route towards standardisation, as large companies pooled their resources into open-source projects for technologies they needed. This increased the chances of broader adoption of such technologies. Today open-source components are present across all layers of the new stack: it’s hard to imagine this stack without them.

Given all this, it is now common to have a combination of technologies (often from different open-source or 3^rd-party vendors) coming together within one application. So the new stack is a lot more heterogeneous than the traditional one.

In a way, significant parts of the new stack can be thought of as a frame rather than fixed architectural entities in themselves. Container technology now enables us to use technologies of our choosing to build parts of the stack and bundle them into containers. These technologies can change over time, but the container abstraction remains. APIs (now spread across all the layers) abstract consumers from the concrete choices made during the implementations, choices that may change in future. Such approaches make the new stack more open to changes than the traditional one.

Conclusion

The new stack represents a radical shift from the traditional one that was common two decades ago. And it solves the problems the traditional stack is unable to, in the context of demands we saw above — demands that originated from internet/mobile/cloud-based applications. The new stack’s distributed nature (coupled with its API centric and ‘software-defined’ approaches) allows applications to scale better (i.e., faster, cheaper, and with greater flexibility), offer higher performance, improved reliability, and better ways to manage huge data volumes. The microservices approach enables applications to evolve parts of the application faster, offer better reliability, and enable large distributed setups to work more efficiently through small teams. The division of the new stack into smaller parts enables it to address niche problem spaces, such as offering databases for specific use-cases or native user interfaces for a given platform — and many of these solutions today come from open source projects that have revolutionised the way the components of this new stack are built.  

But what have been the consequences of this radical shift? What challenges does the new stack bring with it? How will this architecture evolve? Is the traditional stack really obsolete? These are some of the questions we’ll explore in the upcoming articles in this series. A parallel track — written by Bhushan Nigale — will trace the evolution of development and operations practices across these two decades.