Writing¶

How to invest in AI w/ MCPS and Data Analytics

tl;dr: You should build a system that lets you discover value before you commit resources.

!! Key Takeaways

Before asking what to build, start with a simple chatbot to discover what users are interested in. There's no need to reach for a complex agent or workflow before we see real user demand.

Leverage tools like Kura to understand broad user behavior patterns. The sooner we start collecting real user data, the better.

This week, I had conversations with several VPs of AI at large enterprises, and I kept hearing the same story: XX teams experimenting with AI, a CEO demanding results for the next board meeting, sales conference, quarterly review, and no clear path from pilot to production.

These conversations happen because I built improvingrag.com—a resource that helps teams build better RAG systems, which has lead me into many conversations from researchers, engineers, and executives. But the questions aren't about RAG techniques. They're about strategy: "How do we go from experiments to production?" "How do we know what to invest in?" "How do we show ROI?"

Systematically Improving RAG with Raindrop and Oleve

I hosted a lightning lesson featuring Ben from Raindrop and Sid from Oleve to discuss AI monitoring, production testing, and data analysis frameworks. This session explored how to effectively identify issues in AI systems, implement structured monitoring, and develop frameworks for improving AI products based on real user data.

Pricing AI Agents, Headcount, and the Economic Reality

Today I spoke to an executive about SaaS products, and they told me something that shifted my perspective entirely: AI agents need to be compared to budgets that companies draw from headcount, not tooling.

This is one of those insights that seems obvious in retrospect, but completely changes how you think about positioning AI tools in today's market—especially in this era of widespread tech layoffs and economic uncertainty.

There Are Only 6 RAG Evals

The world of RAG evaluation feels needlessly complex. Everyone's building frameworks, creating metrics, and generating dashboards that make you feel like you need a PhD just to know if your system is working.

Free Lightning Lessons to Advance Your RAG Implementations

I'll be hosting industry experts to share practical techniques for enhancing your Retrieval Augmented Generation (RAG) systems.

Creating Content That Converts: My Guide for AI Consultants

Note: this post is archival. I’m no longer taking consulting engagements or running cohorts.

Why I Prioritize Content (And Why You Should Too)

Let me share something I wish I'd understood sooner: consistent content creation isn't just a marketing tactic—it's the foundation of a thriving consulting business.

When I started my consulting journey, I was stuck in the time-for-money trap. I'd jump on Zoom calls with prospects, explain the same concepts repeatedly, and wonder why scaling was so difficult. Then I had a realization that changed everything: what if I could have these conversations at scale?

Now I extract blog post ideas from every client call. Every Friday, I review about 17 potential topics from the week's conversations. I test them with social posts, see which ones get traction (some get 700 views, others 200,000), and develop the winners into comprehensive content.

Here's why this approach has transformed my business:

Version Control for the Vibe Coder (Part 1)

Imagine this: you open Cursor, ask it to build a feature in YOLO-mode, and let it rip. You flip back to Slack, reply to a few messages, check your emails, and return...

It's still running.

What the hell is going on? .sh files appear, there's a fresh Makefile, and a mysterious .gitignore. Anxiety creeps in. Should you interrupt it? Could you accidentally trash something critical?

Relax—you're not alone. This anxiety is common, especially among developers newer to powerful agents like Cursor's. Fortunately, Git is here to save the day.

Version Control for the Vibe Coder (Part 2)

In Part 1, you learned the basics of safely using Git with Cursor agents. Now, let's level up your workflow by diving into advanced Git practices and explicitly instructing Cursor to handle these for you.

Fine-Tuning Embedding Models for Enterprise RAG: Lessons from Glean

Retrieval-Augmented Generation (RAG) systems have become essential tools for enterprises looking to harness their vast repositories of internal knowledge. While the theoretical foundations of RAG are well-understood, implementing these systems effectively in enterprise environments presents unique challenges that aren't addressed in academic literature or consumer applications. This article delves into advanced techniques for fine-tuning embedding models in enterprise RAG systems, based on insights from Manav Rathod, a software engineer at Glean who specializes in semantic search and ML systems for search ranking and assistant quality.

The discussion focuses on a critical yet often overlooked component of RAG systems: custom-trained embedding models that understand company-specific language, terminology, and document relationships. As Jason Liu aptly noted during the session, "If you're not fine-tuning your embeddings, you're more like a Blockbuster than a Netflix." This perspective highlights how critical embedding fine-tuning has become for competitive enterprise AI systems.

Authority in RAG Systems: The Missing Piece in Your Retrieval Strategy

Here's the thing about RAG (Retrieval-Augmented Generation): everyone's obsessed with fancy embeddings and vector search, but they're missing something crucial – authority matters just as much as relevance.

My students constantly ask about a classic problem: "What happens when new documents supersede old ones?" A technical guide from 2023 might be completely outdated by a 2025 update, but pure semantic search doesn't know that. It might retrieve the old version simply because the embedding is marginally closer to the query.

This highlights a bigger truth: relevancy, freshness, and authority are all critical signals that traditional information retrieval systems juggled effectively. Somehow we've forgotten these lessons in our rush to build RAG systems. The newest and shiniest AI technique isn't always the complete solution.

I've spent years working with ML systems, and I've seen this pattern repeatedly. We get excited about semantic search, but forget the hard-won lessons from decades of information retrieval: not all sources deserve equal trust.