For most of my career as a VP of Engineering, navigating that messy transition was my core job. My leadership teams and I relied heavily on Tuckman’s classic framework—Forming, Storming, Norming, and Performing—to track milestones and build high-performance engineering cultures.

Lately, I’ve been getting a massive sense of déjà vu.

At EGI (egi.sh), we’ve been deploying autonomous agents into heavy-lifting growth and operational roles across various industries. As we track the real-world data of how these human-agent teams interact, a striking truth has emerged:

Agent adoption is emotional before it is operational.

We like to treat AI as a software deployment—like upgrading a database or swapping an API. But the moment you give an agent autonomy over a live workflow, it ceases to be a tool. It becomes a teammate. And human teams react to silicon teammates exactly the way they react to new human ones.

If you are introducing AI agents to your organization, you aren’t just managing code; you’re managing group psychology. Here is how the four stages actually play out in the trenches.

1. Forming: Curiosity and Fear

• The Vibe: Guarded fascination.

• The Human Internal Dialogue: “What is this thing? Can I trust it? Is it here to help me, or am I training my replacement?”

When you first introduce an agent to a workflow, management is usually thrilled. The team, however, goes quiet. Because the technology is unproven to them, humans project their extremes onto it: they either assume it’s a magic bullet that will do all their work, or they completely ignore it, hoping the pilot program dies.

The Goal: Familiarity. Your job here isn’t to prove the agent is perfect. It’s to demystify it. At EGI, we’ve found the quickest way through this is defining strict boundaries—letting the team know exactly what the agent is tasked with, and exactly where its guardrails end.

2. Storming: Friction and Skepticism

• The Vibe: Frustration and “I told you so.”

• The Human Internal Dialogue: “Why did it do that? The output is wrong. I could have done this manually in five minutes. This is wasting my time.”

Storming starts the exact minute the agent touches real production data. The honeymoon ends, and reality hits. The agent misinterprets a nuanced customer ticket, hits a legacy API rate limit, or runs into an uncodified edge case.

This is the exact valley where most corporate AI initiatives quietly go to die. Teams mistake the natural friction of onboarding for a fundamental failure of the technology.

[Forming: The Demo] ➔ [Storming: Edge Cases & Misfires] ➔ [The Trap: Pulling the Plug]

The Goal: Trust through correction. You wouldn’t fire a human junior engineer for making a mistake on day three; you’d give them feedback. Agents require the same grace. Storming is resolved through continuous calibration loops. Leaders need to normalize these early errors as part of the integration cost.

3. Norming: Trust and Ritual

• The Vibe: Cautious confidence.

• The Human Internal Dialogue: “What should I delegate to it today? When do I need to step in? How do we hand things off?”

During norming, the team stops fighting the agent and starts adapting to it. This is where real operating rhythms are born.

Humans figure out the agent’s specific strengths and quirks. They build new daily rituals, review loops, and habits around its output. The dynamic shifts from adversarial (“me vs. the machine”) to collaborative.

The Goal: Predictable flow. This is where you solidify your Human-in-the-Loop (HITL) frameworks. You define the precise hand-off points where machine execution ends and human intuition takes over.

4. Performing: Leverage and Flow

• The Vibe: True leverage.

• The Human Internal Dialogue: “How did we ever run this department without this? What else can we hand over to it?”

At the performing stage, the agent becomes part of the team’s muscle memory. It’s no longer an “AI project” or an experimental tool—it’s just how work gets done.

The human operators stop acting as rote doers of repetitive tasks and step into their true role as editors and strategic orchestrators. At this point, team output decouples from linear headcount.

The Management Mistake Moving Forward

The biggest mistake I see leadership teams making right now is expecting an agent to jump straight from Forming to Performing. They sign a SaaS vendor contract, run a cool demo, and expect a 10x spike in operational efficiency by next Friday.

When the team inevitably hits the Storming phase—because all real-world deployments do—leadership panics, assumes the product is broken, and abandons it.

Friction isn’t a sign that the technology failed; it’s a mandatory milestone on the path to high performance. If you want to scale an organization today, you have to build a culture that knows how to onboard silicon just as effectively as it onboards flesh and blood.

We’re seeing these dynamics unfold every day across the teams we collaborate with at egi.sh. If you’re currently introducing agents to your operations, which stage is your team stuck in? Let’s talk in the comments.

.

I wrote my first machine learning code nearly 25 years ago in a tiny Evolutionary Computation lab.

Back then, we were obsessed with the “stochastic”—using randomness and selection to evolve solutions. It felt like magic. But I never expected the world to come full circle: Moving away from probabilistic “guessing” back toward the rigor of hard-coded execution for the world’s most complex business problems.

As we head into 2026, the era of “vibes-based AI” is officially over.

The 2025 Realization: RAG hit a ceiling.

At the start of this year, the industry was blinded by “Agentic RAG.” The assumption was that if we just gave a model enough “context,” it could “reason” its way through anything.

But our users—a group of the most innovative CEOs, CROs, and Growth leaders—pushed us past that. They brought us problem after problem that simply wasn’t solvable via retrieval.

They didn’t need a summary of their data; they needed to trust the AI to run their business logic. They realized what we now know: You cannot “reason” your way to a correct balance sheet. You have to compute it.

The Shift: Neuro-Symbolic Execution

Driven by that customer demand, we spent 2025 perfecting an architecture that treats the LLM as a Compiler, not a database.

We built the bridge we always imagined “General Intelligence” would bring:

• The Connection: Real-time, secure hooks into CRMs, ERPs, and Checkout systems.

• The Action: The system writes and executes Python code on the fly to solve the problem.

• The Result: Deterministic truth. No inference. No “hallucinating” the revenue.

Compile → Compute → Explain

Our 2025 breakthrough was defining these boundaries:

1. Compile: Natural language is turned into executable logic.

2. Compute: That code runs against live systems (Salesforce, SAP, Stripe).

3. Explain: Only then does the LLM narrate the “why” and the “what next.”

Looking Toward 2026

After two and a half decades in AI, the lesson of 2025 is the clearest one yet:

• Retrieval is for context.

• Execution is for truth.

• Language is for the interface.

To the leaders who pushed us to build this: thank you for refusing to settle for “close enough.” You forced us to move from “seeing if it works” to “ensuring it’s right.”

If 2025 was the year the industry realized the limits of the stochastic approach, 2026 will be the year of the Deterministic Agent.

But how does it actually think? How can it juggle dozens of complex product activities, from grooming a backlog to running user research, as if it were a seasoned human PM?

The secret lies in its architecture. In this post, we’ll pull back the curtain and show you how Alfred uses LangGraph, an open-source framework for building stateful, agentic systems. We'll explore why LangGraph is the key to unlocking Alfred's autonomy and how it allows our AI PM to manage complex product workflows from end to end.

The Illusion of Autonomy: Why Single Prompts Fail at Product Management

Large language models (LLMs) are revolutionary for one-off tasks like summarizing text or answering a question. But product management is not a series of one-off tasks. It's a continuous, dynamic loop. A human PM must:

• Synthesize information from disconnected sources like user interviews, support tickets, and analytics dashboards.

• Make strategic trade-offs based on evolving company goals and resource constraints.

• Create and iterate on detailed specifications, tickets, and roadmaps.

• Coordinate work across engineering, and go-to-market teams.

• Constantly adjust the plan as new data and feedback arrive.

This requires long-running, stateful reasoning, not just a single prompt-and-response. Chaining simple LLM calls together is brittle and quickly loses context. To achieve true autonomy, an AI needs a more robust operating system.

Why LangGraph? Building a Mind That Can Reason and Act

LangGraph is designed to overcome the limitations of single-prompt systems. It allows us to build AI agents that can plan, remember, and adapt over time using graphs. For Alfred, this provides a powerful foundation built on a few key capabilities:

• Graph-Based Workflows: Alfred’s tasks are mapped as nodes in a graph (e.g., ingest_feedback, generate_insights, draft_prd). LangGraph manages the transitions between these nodes based on rules, dependencies, and outcomes. This creates a resilient, repeatable process for complex work.

• Persistent Memory: Alfred retains context across all its interactions. When asked to "summarize Q3 feedback and update the roadmap," it doesn't start from scratch. It executes a multi-step workflow—analyzing qualitative data, comparing it to the existing roadmap, re-prioritizing items, and generating a summary—all while preserving the state and intermediate data.

• Agentic Tool Use: Alfred does more than just generate text. LangGraph enables it to be an actor. It can call external APIs, run internal data analysis tools, create a ticket in Jira, or generate a diagram with a dedicated service. Each action is a tool it can decide to use.

• Dynamic Branching and Adaptation: Product management is unpredictable. Based on new information, Alfred can dynamically alter its course. It might pause to ask a clarifying question before writing a spec, or skip a roadmap update if there's no new evidence to support a change. This allows it to reason and react, not just follow a rigid script.

Anatomy of an Autonomous PM: Alfred in Action

Here’s how Alfred leverages LangGraph to perform core product management functions:

1. From Noise to Signal: Product Discovery

When you ask, "What are the top three issues our enterprise customers faced this month?" Alfred initiates a discovery graph:

• Node 1: Ingest Data: It pulls in call transcripts, support tickets, and survey results from various sources.

• Node 2: Cluster & Summarize: It uses analysis tools to identify recurring patterns and themes within the raw data.

• Node 3: Cross-Reference: It checks these findings against current roadmap priorities and business goals.

• Node 4: Synthesize & Report: It returns a concise summary of insights and can even suggest next steps, like drafting a problem brief.

2. From Idea to Ticket: Product Definition

Alfred transforms high-level ideas into actionable engineering work:

• It takes a raw input, like "Users are struggling to reset their passwords."

• It invokes a specification_generator node to draft acceptance criteria, define edge cases, and outline success metrics.

• Using its tool integrations, it then calls a create_jira_ticket node, populating it with the generated spec.

• Finally, it can trigger notifications to design and QA systems, ensuring team-wide alignment.

3. From Plan to Progress: Execution & Monitoring

Alfred doesn’t just create tickets; it tracks them to completion:

• It periodically queries engineering APIs to check the status of in-flight projects.

• If it detects a blocker or a stalled ticket, it can alert the relevant stakeholders.

• It can even suggest corrective actions, like splitting a large story or re-prioritizing work to meet a deadline.

LangGraph as Alfred’s “Brainstem”

If Alfred's "higher intelligence" is its ability to strategize and reason, then LangGraph is its brainstem—the fundamental system that coordinates its core functions without conscious effort. It provides:

• State Control: Maintaining constant awareness of which tasks are running, pending, or complete.

• Action Dispatch: Deciding what to do next, whether it's analyzing data or calling an API.

• Parallel Processing: Running multiple workflows simultaneously—like monitoring competitive intel while clustering user feedback—without losing coherence.

This architecture is what makes Alfred feel like a true digital teammate rather than a collection of disconnected AI gadgets.

The Road Ahead: Towards Proactive and Collaborative AI

With a stateful foundation like LangGraph, Alfred is evolving from a reactive assistant to a proactive leader. It can continuously monitor signals like churn spikes or drops in feature adoption and independently recommend interventions, just as a senior product leader would.

We envision a future where Alfred can run automated design-test-redesign loops, collaborate with other specialized agents (like "Bruce," the autonomous sales engineer), and run real-time experiments to test product hypotheses.

Conclusion

Alfred is more than an LLM in a product manager's costume; it's an autonomous agent with a scalable, resilient brain. LangGraph provides the backbone for it to manage complex, stateful, and dynamic workflows from end to end.

For anyone building AI systems meant to do real, multi-step work, moving beyond one-shot prompts is essential. Frameworks like LangGraph bridge the critical gap between language generation and true autonomous execution, unlocking the next generation of intelligent systems.

As an engineering leader, I’ve always had a strong bias toward building in-house. I believed that building gave us more control, more flexibility, and ultimately a better product tailored to our needs. Until recently, that conviction held true.

After over six months of building production-grade agents, though, my perspective has shifted. Building agents is a far more complex endeavor than it might seem on paper. In this post, I’ll break down why.

Evals: The Invisible Heavy Lift

When you set out to build an agent, you quickly realize that creating robust evaluation frameworks (evals) is critical. Evals act as your performance benchmarks, guiding you in measuring whether your agent is truly doing what you designed it to do.

Surprisingly, building solid evals often consumes about 25% of the entire effort. Without them, there is no reliable way to ensure that your agent’s behavior aligns with expectations, especially when it starts handling real-world edge cases. Evals require continuous iteration, careful scenario design, and meticulous tracking. And they are only the beginning.

The Complexity of Model Orchestration

Gone are the days when you could rely on a single “best” large language model (LLM) to cover every use case. Today, each model, whether from OpenAI, Anthropic, Google, or others, has distinct strengths and trade-offs.

Choosing and orchestrating multiple models to work together seamlessly is far from trivial. You need to evaluate which model excels at which task, route requests intelligently, and continuously monitor performance as new models and updates roll out.

This constant evolution introduces another layer of operational overhead. You have to keep up with the rapid velocity of new model launches, each with different performance characteristics and cost implications.

Functions Aren’t as Unique as We Think

We often convince ourselves that our business processes are so unique that only a custom-built agent can handle them. In practice, most businesses operate within well-established best practices of their industry.

This means that the “functions” we think set us apart are often not that different at all. Reinventing the wheel to accommodate slightly off-norm workflows is rarely justified and usually adds unnecessary complexity and maintenance burden.

Buying or leveraging an existing platform can help enforce discipline in sticking to standards, which often improves overall operational efficiency.

The Real Cost: Opportunity

Perhaps the most critical and overlooked factor is opportunity cost.

When we allocate engineering resources to building internal agent infrastructure, we divert time and focus away from growth initiatives. These are the very features and improvements that drive product adoption and revenue.

Every hour spent refining evals, fine-tuning models, or maintaining internal agent tooling is an hour not spent shipping new customer-facing capabilities or closing deals. In a rapidly evolving market, this trade-off can significantly affect your competitive position.

Conclusion

Building agents from scratch can feel like the ultimate expression of technical ownership. But after living through the reality of endless eval cycles, complex model orchestration, constant maintenance, and the slow erosion of focus from growth priorities, I have come to see the value of buying or adopting a mature solution.

It is not just about reducing technical burden. It is about making better strategic bets and preserving precious engineering capacity to invest where it truly differentiates your business.

If you are on the fence, I encourage you to weigh these hidden costs carefully. You might find that “buy” is not just the easier path, it is the smarter one.

But there’s a dark side to this relentless focus on forward momentum: every feature we launch has a shadow, a set of unintended consequences that we ignore at our peril.

Your Brilliant Feature Might Be a Ticking Time Bomb

Think your new feature is a guaranteed win? Let’s talk about that AI-powered support agent you’re so excited about. You sold it as a marvel of efficiency that would slash support costs.

But what if it’s also a trust-destroying machine? What if it confidently hallucinates answers that get your company in legal trouble? Or what if it’s so cold and unhelpful that it makes your most loyal customers feel like a line item on a spreadsheet?

Your cost-saving feature just became your churn-creating nightmare. This is the central paradox of product development: an idea’s potential for good is often directly proportional to its potential for harm.

Negligence Is Not an Excuse

"We didn't see it coming" is no longer a valid excuse. Ignoring the potential for misuse, abuse, and collateral damage is a failure of leadership. This negligence has real costs:

• Trust: Once broken, it’s nearly impossible to rebuild.

• Reputation: Your brand can be permanently tainted by a feature that enables harassment, spreads misinformation, or exploits vulnerable users.

• Ethics: Are you building tools that make the world better, or just more efficient at being worse?

Your job isn't just to ship. It's to be the first line of defense for your users and your company's integrity.

How to See Around Corners

Anticipating the negative isn't about negativity; it's about professional diligence. Here’s how to make it part of your DNA:

1. Run a "Premortem," Not a Postmortem. Before a single line of code is written, gather your team and ask: "How does this feature fail in the most epic way possible?" Encourage brutal honesty. To take this a step further, I use AI as my dedicated sparring partner. I'll prompt an agent like Alfred to become my worst nightmare: a bad actor trying to exploit the feature, a competitor looking for weaknesses, or a cynical user determined to misunderstand it. This form of adversarial thinking forces out the ugly possibilities before they become reality.

2. Assume Your Users Aren't Saints (or Experts). Design for reality. How will a troll abuse this feature? How will a confused user break it? How could it be weaponized by a bad actor? Build for the worst-case user, not just the best-case one.

3. Install Circuit Breakers. Don't launch a powerful tool without an off-switch. Build in guardrails from day one: rate limits, clear disclaimers, user-controlled privacy settings, and dead-simple "escape hatches" to talk to a real human.

4. Treat Launch Day as Day Zero. Your job isn't done when the feature is live. It's just started. Watch the data like a hawk—not just the vanity metrics, but the support tickets, the social media complaints, and the weird usage patterns. Be humble enough to admit when you got it wrong and agile enough to fix it fast.

Stop Building Fragile Products

Building with foresight isn't about adding bureaucracy. It’s about forging antifragile products that can withstand the chaos of the real world. It’s the difference between being a feature factory and being an institution that people trust.

So, the next time you write a PRD, stop and think. What's the worst thing someone could do with this? And how do we stop them?

Good PMs build what’s on the roadmap. Great PMs build what lasts.

Use it during quarterly planning, team reviews, or when something feels off and you want clarity.

What It Measures

AINI breaks down “AI-native” into five parts. For each one, score yourself 1 to 5.

1. Data Foundation
Do you have clean, structured data that AI can use?

2. AI Adoption
Is meaningful work being done by or with AI — not just tooling, but real contribution?

3. Workflow Integration
Is AI part of the team’s daily systems and triggers, or just bolted on the side?

4. Cultural Fluency
Does your team know how to use AI well, and do they push its limits?

5. Strategic Leverage
Is AI helping you ship faster, cut costs, raise margins, or unlock things you couldn’t do before?

How to Score It

Each of the five dimensions has a weight:

• Data Foundation: 20%

• AI Adoption: 25%

• Workflow Integration: 20%

• Cultural Fluency: 15%

• Strategic Leverage: 20%

Give yourself a score from 1 to 5 in each category. Multiply by the weight. Add everything up. You’ll get a final score between 0 and 5.

What Your Score Means

0 to 1.4 — Manual-first
Fully human-driven. No infrastructure in place.

1.5 to 2.9 — Augmented
Some AI use, but it's not core to how the team operates.

3.0 to 3.9 — Embedded
AI is supporting core workflows. It’s part of how the team gets work done.

4.0 to 5.0 — Autonomous
AI is essential. The function runs on it.

Example: Sales

Data Foundation: 5 → 1.0
AI Adoption: 5 → 1.25
Workflow Integration: 4 → 0.8
Cultural Fluency: 4 → 0.6
Strategic Leverage: 5 → 1.0
Total: 4.65 out of 5 → Autonomous

Sales is likely using AI agents to write outreach, summarize calls, coach reps, and forecast pipeline. It’s foundational and improving over time.

Example: Engineering

Data Foundation: 3 → 0.6
AI Adoption: 2 → 0.5
Workflow Integration: 2 → 0.4
Cultural Fluency: 2 → 0.3
Strategic Leverage: 2 → 0.4
Total: 2.2 out of 5 → Augmented

Engineering may have GitHub Copilot enabled or some internal scripts, but AI isn’t central to how the team builds or ships product yet. Still early.

When to Use It

• Quarterly planning

• Team offsites

• 1:1s with function heads

• Product reviews

• Anywhere you need a gut check on AI progress

Download the Template

I put together a simple Excel/Sheets version of the AINI.

📥 Download it here:
AI Nativity Index Scoring Template

Includes:

• Editable scoring rows

• Weights and formulas pre-set

• Easy to duplicate across teams or quarters

Final Thought

You don’t need a 10-page AI strategy.
You need to know where AI is actually changing how your team works — and where it’s not.

This tool gives you the signal.

Prompt engineering is the methodical practice of designing inputs that elicit useful, targeted responses from language models. Based on OpenAI’s Prompting Guide for GPT-4, here are five specific techniques product managers can apply to use ChatGPT more effectively.

1. Be Specific: Provide Scope and Structure

Ineffective prompt:
"Summarize customer feedback."

Improved version:
"Summarize the top three recurring problems mentioned in this feedback. For each, include the user persona, relevant product area, and one representative quote."

Rationale: Specific prompts reduce ambiguity and help the model focus on actionable insights.

2. Provide Examples for Calibration

If you're looking for structured responses, show ChatGPT what good looks like.

Prompt:
"Format feedback as shown:

• Input: 'App crashes during login'

• Output:

  • Persona: IT Admin

  • Theme: Stability

  • Sentiment: Negative

  • Quote: 'App crashes during login.'

Now, do the same for the following feedback set."

Rationale: Examples work as informal training, guiding the model toward your preferred structure and style.

3. Ask for Structured Output

Instead of generating long-form text, request formatted output that's ready to use in documentation.

Prompt:

For each theme identified in the feedback, format the output as follows:

## Problem Theme: <Theme>
- Personas: <List>
- Quotes:
  - "<Quote 1>"
  - "<Quote 2>"
- Frequency: <X out of Y instances>

Rationale: Structured responses are easier to scan, compare, and incorporate into product docs and planning materials.

4. Define ChatGPT's Role

Assigning a role to ChatGPT aligns its responses with your expectations.

Prompt:
"You are a product analyst reviewing customer feedback from school districts. Identify user pain points, hypothesize potential root causes, and propose relevant product areas."

Rationale: Role-based prompts influence tone, analytical depth, and response format.

5. Treat Prompts Like Product Iterations

Prompt engineering benefits from the same mindset as feature development: iteration and testing.

Approach:

• Begin with a draft prompt

• Test on a small dataset

• Adjust: Add constraints, clarify language, change structure

Rationale: Continuous refinement helps surface better insights, reduce hallucinations, and tailor output to your workflow.

Summary

Just like software engineering, prompt engineering helps you get the most out of a system. Learning to engineer it well could become a key differentiator in your PM Craft.

Introduction

In today's rapidly evolving business landscape, making effective decisions at scale is more critical than ever. Psycholinguistics—the study of how language and cognitive processes interact—offers valuable insights into why artificial intelligence (AI) is increasingly suited to support and, in some cases, make business decisions. Decisions in business often fall into two categories: Type 1 (high-stakes, irreversible) and Type 2 (low-stakes, reversible). The vast majority of decisions are Type 2, and if not made efficiently, they can significantly slow down an organization.

AI is particularly well-suited to handle Type 2 decisions due to its ability to process patterns, heuristics, and language at speeds that humans cannot match. These decisions typically involve clear data, repetitive patterns, and straightforward feedback loops—conditions in which AI thrives. Type 1 decisions, which require more deliberate thought, creativity, and ethical considerations, can still benefit from AI acting as a data-driven partner, although they ultimately require human judgment.

Type 1 vs. Type 2 Decisions

Type 1 Decisions

• Definition: High-stakes, irreversible decisions that require careful deliberation.

• Examples: Mergers and acquisitions, major capital investments, or hiring key executives.

• Cognitive Process: Align with Daniel Kahneman's System 2 thinking—slow, analytical, and deliberate.

Type 2 Decisions

• Definition: Low-stakes, reversible decisions that can be adjusted if necessary.

• Examples: Adjusting project priorities, modifying team goals, or optimizing logistics.

• Cognitive Process: Correspond to System 1 thinking—fast, automatic, and intuitive.

From a psycholinguistic perspective, AI's ability to process language and patterns rapidly positions it well for handling Type 2 decisions. These are decisions where speed and efficiency outweigh the need for deep, contextual understanding.

How AI Leverages Psycholinguistics for Type 2 Decisions

Pattern Recognition and Heuristics

Humans often rely on heuristics—mental shortcuts based on pattern recognition—to make quick decisions. Psycholinguistically, we're adept at processing language cues to interpret information rapidly. AI models, particularly those using machine learning algorithms, excel at detecting patterns within large datasets.

Example: In e-commerce, AI analyzes customer behavior to make real-time product recommendations. Amazon's recommendation engine, which contributes to a significant portion of its sales, exemplifies AI making countless Type 2 decisions per second.

Natural Language Processing in AI

Natural Language Processing (NLP) enables AI to understand and generate human language, mirroring aspects of human psycholinguistic processing such as syntax parsing and semantic understanding.

Example: AI can use NLP to interpret queries and provide relevant responses, improving efficiency and customer satisfaction. Tools like IBM Watson Assistant help businesses automate interactions while maintaining a conversational tone.

Learning from Data Exposure

Just as humans improve decision-making with experience, AI models enhance their accuracy through exposure to more data. However, AI can process and learn from vast datasets far beyond human capacity.

Example: PayPal's fraud detection system employs machine learning to analyze millions of transactions, identifying fraudulent activity with high accuracy and reducing the fraud rate below industry averages.

AI as a Thought Partner for Type 1 Decisions

While AI may not autonomously make high-stakes Type 1 decisions, it serves as a valuable assistant by providing data-driven insights that inform human judgment.

Data Analysis and Risk Assessment

AI can process complex datasets to identify trends, risks, and opportunities that might not be immediately apparent to human analysts.

Example: Investment banks use AI algorithms to analyze market data and predict financial trends, assisting in strategic decisions like mergers and acquisitions. Goldman Sachs, for instance, utilizes AI to enhance its analytical capabilities in evaluating potential deals.

Simulation and Scenario Planning

AI models can simulate outcomes based on various inputs, helping decision-makers evaluate potential consequences without real-world risks.

Example: In supply chain management, AI tools forecast demand and simulate logistics scenarios, enabling companies to optimize operations before implementing changes.

Cognitive Limitations and Human Bias

Mitigating Cognitive Biases

Human decisions are susceptible to biases such as confirmation bias or overconfidence. AI, when designed correctly, can offer objective analyses that counteract these tendencies.

Example: In recruitment, AI platforms like Applied use anonymized data to focus on candidate skills and experiences, reducing unconscious biases in hiring processes.

Risks of AI Bias

It's crucial to acknowledge that AI can inherit biases present in training data. Continuous monitoring and updating of AI models are necessary to maintain fairness and accuracy.

Example: Facial recognition systems have faced criticism for higher error rates among certain demographics, highlighting the need for diverse and representative training data.

Ethical Considerations and Industry Implications

Healthcare

In healthcare, AI can aid in diagnostics and treatment recommendations but must be used cautiously due to ethical implications.

Finance

AI can assiss in fraud detection and investment strategies but requires transparency to maintain trust.

Implementation Challenges and Solutions

Data Quality and Availability

• Challenge: AI's effectiveness depends on high-quality data, which may be fragmented or inconsistent.

• Solution: Invest in data infrastructure and governance to collect, clean, and maintain data integrity.

Integration with Existing Systems

• Challenge: Legacy systems may not support AI technologies.

• Solution: Adopt scalable platforms and APIs that enable gradual integration without overhauling existing systems.

Employee Adoption

• Challenge: Resistance due to fear of job displacement or distrust in AI decisions.

• Solution: Provide training and involve employees in AI implementation to enhance acceptance and collaboration.

Regulatory Compliance

• Challenge: Navigating complex regulations related to data privacy and AI use.

• Solution: Engage legal experts early in the process and design AI systems with compliance in mind.

Practical Recommendations

1. Start with Clear Objectives: Identify specific business areas where AI can add immediate value, particularly in Type 2 decision-making processes.

2. Ensure Ethical AI Practices: Develop guidelines that address bias, transparency, and accountability in AI systems.

3. Invest in Human-AI Collaboration: Encourage teams to view AI as a tool that enhances their capabilities rather than a replacement.

4. Monitor and Iterate: Continuously assess AI performance and make adjustments based on feedback and changing business needs.

5. Educate Stakeholders: Communicate the benefits and limitations of AI to all stakeholders, fostering an environment of informed adoption.

Conclusion

AI is increasingly capable of supporting business decision-making, especially for low-stakes, reversible Type 2 decisions. Its ability to process language and patterns rapidly, coupled with continual learning from vast datasets, makes it an invaluable asset for modern organizations.

In high-stakes Type 1 decisions, AI serves as a powerful assistant, providing data-driven insights that enhance human judgment. However, the irreplaceable human elements of empathy, creativity, and ethical reasoning remain essential.

Successful integration of AI into business processes requires thoughtful implementation, attention to ethical considerations, and a focus on augmenting rather than replacing human capabilities. By embracing AI as a partner in decision-making, businesses can achieve greater efficiency, innovation, and competitive advantage.

This multi-modal talent force will have common responsibilities and performance evaluation criteria, ensuring fairness and transparency.

The best way for humans to prepare for and feel fair about coexisting with our new AI peers is to ensure that these peers are held to the same performance benchmarks as us. Then, it is a fair fight for excellence.

What Leading Indicators Drive My Coexistence Hypothesis?

I see large language models (LLMs) writing more effective code from a testability, readability, and maintainability POV than I did three years out of college. In most of my daily coding tasks, I now play the role of a reviewer/architect, adding strategic value. This reminds me of the work I used to do for architects and principal engineers earlier in my career. This trend is not restricted to engineering discipline; I see similar patterns emerge in other, non-software engineering job functions as well.

What Gives Me the Authority to Benchmark My AI Peers?

Career Ladder Growth: I grew from a Software Engineering Intern to a Vice President of Engineering. My first task was to fix an inconsequential bug in C#. My current tasks impact millions of dollars in revenue. I’ve seen firsthand the unique performance objectives of each level of the ladder.

Sample Size: The first production code I shipped was in 2004, 20 years ago. These two decades have given me a sample size of millions of lines of code (LOC), thousands of human employee peers, and hundreds of business scenarios in both enterprises and startups.

Vantage Point: Engineering executives have a unique vantage point. They are exposed to the performance criteria of software engineers and non-technical stakeholders, from GTM to Ops. This unique perspective is crucial in understanding AI-peer coexistence in enterprise workforces.

What Will Be My Benchmarking Criteria?

I will keep it simple. Complexity hurts understandability. The goal is to develop understanding in pursuit of harmonious and force-multiplying coexistence.

The Rule of Threes:

For each organizational role (Engineering, GTM, Ops) that a human employee currently holds:

1. Pick three job responsibilities

2. Pick three levels of performance ratings

3. Pick three feedback sources (self, manager, and peers)

I will then have AI employees go through the 6-month performance review cycle for their respective roles.

I will stop here. You can read more about the benchmarks by joining The Outlier Engineer

BTW, if you have a specific function that you would like to be evaluated against its AI-employee peer, please DM me.

What is NOT Customer-Obsession?

1. ELT-ing TBs of Customer Data: Extract, Load, and Transform (ELT) processes that handle terabytes of customer data are necessary, but simply managing data doesn’t equate to understanding or solving customer problems.

2. Shipping a Bug-Free Model: While delivering a model free of bugs is essential, it is a basic expectation. A bug-free product that doesn’t address customer needs is still a failure.

3. Deploying the Most Cutting-Edge Models: Utilizing state-of-the-art models is exciting and can be beneficial, but it is not customer-obsession. If these models don’t translate into real value for the customer, they miss the mark.

Time and again, we have all come across expensive ML/AI initiatives that deliver zero to minimal value to customers. One common factor in many of these initiatives is the lack of customer-obsession

What is Customer-Obsession in the Context of ML/AI?

There are three pillars of effective customer-obsession:

1. Enable

2. Champion

3. Elevate

Using an example of an ML-AI native CRM, let's expand on these pillars. For simplification, let's consider three distinct user personas:

1. VP Sales

2. Account Executive (AE)

3. Sales Development Representative (SDR)
   
   Each of these personas will have their unique needs for enabling, championing, and elevating.

Enabling: Reducing Customers’ Jobs-to-be-Done

• VP Sales: Enable VPs by providing predictive models that help in strategic decision-making, like quota prediction and team OTE confidence scores.

• AE: Enable AEs by automating tasks like personalized positioning and lead scoring, allowing them to focus more on strategic activities.

• SDR: Enable SDRs by equipping them with tools that streamline predictive lead generation and personalized audiences, making it easier to identify and engage with potential customers.

Championing: Supporting Customers’ Performance KPIs

• VP Sales: Champion their goals by ensuring the CRM provides accurate and actionable recommendations that align with their key performance indicators (KPIs), such as pipeline growth, forecast optimization, and customer acquisition cost optimization.

• AE: Champion AEs by providing tools that automatically prioritize leads based on their performance against quotas, conversion prediction, and sales cycle optimization, helping them to optimize their on-target earnings (OTEs).

• SDR: Champion SDRs by offering insights into their outreach effectiveness, tracking metrics such as response rates, number of meetings scheduled, and lead conversion prediction, lead classifiers, allowing them to refine their approach and improve outcomes.

Elevating: Empowering Customers for High-Leverage Activities

• VP Sales: Elevate VPs by freeing up their time from operational details, enabling them to focus on high-leverage activities like strategic planning, team leadership, and market expansion.

• AE: Elevate AEs by reducing administrative burdens, allowing them to dedicate more time to building relationships, negotiating with clients, and closing deals.

• SDR: Elevate SDRs by automating repetitive tasks, enabling them to focus on crafting personalized messages, engaging more meaningfully with prospects, and improving their overall productivity.

Conclusion

While understanding the mathematical intricacies of ML/AI is important, the true measure of success lies in our ability to obsess over our customers. By centering our efforts on their needs, KPIs, and overall success, we can ensure that our ML/AI initiatives deliver real value and drive significant outcomes.

How are you ensuring customer-obsession in your ML/AI initiatives? Share your thoughts and experiences in the comments below!

Over the years, I've observed various ML methodologies being applied with differing levels of effectiveness and transparency across business sectors. Below, I highlight the enduring factors that underscore the relevance of Trad-ML.

The Continuing Relevance of Trad-ML:

Model Performance:
Problem-focused solutions inherent in Trad-ML exhibit a natural elegance. Whether it's making predictions, classifying data, providing recommendations, interpreting natural language, or clustering datasets, these tasks are ubiquitous across sectors like finance, human resources, and healthcare. Trad-ML not only offers time-tested models and established benchmarks ensuring a base level of effectiveness but is also supported by robust open-source frameworks like PyTorch, TensorFlow, and SciKit-Learn. These frameworks and benchmarks empower even non-specialist full-stack software engineers to achieve baseline standards of model performance effortlessly.

Model Explainability:
The necessity for humans to understand the reasoning behind machine-based decisions becomes critical when these decisions impact areas such as health, finance, and employment. Trad-ML, supported by well-established ML-Ops practices, enhances the auditability and interpretability of both data-driven and model-driven parameters. This is especially important as AI safety standards like ISO 42001 place a premium on the explainability provided by the datasets used in ML operations.

Model Costs:
The high cost of AI-specific, purpose-built GPU/TPU architectures often makes them prohibitive for many businesses, thus stifling innovation at the grassroots level. In contrast, most Trad-ML techniques can be implemented using generic, widely available GPU/CPU architectures. This compatibility means that most existing cloud and personal computing resources are already suitable for training and deploying Trad-ML models, significantly simplifying financial approval processes for CFOs and tech leaders.

Acknowledging the Impact/Unlock of Gen-AI:
The most profound influence of Gen-AI in the business and technical realms is its role in elevating ML/AI from a secondary optimization tool to a primary implementation approach. This shift, driven by the momentum of Gen-AI, has established both traditional and generative machine learning as pivotal elements in technological deployments.

Additionally, there are also problem domains specifically suited for Gen-AI. I will cover them in detail and approaches to measure there effectiveness in separate article. Stay tuned!