>_faiz.palwala

The Dogfood Incident: What Happened When Beta Testers Fed LinkedIn and GitHub into My Ingestion Engine

October 19, 2026[Tech]

The quickest way to humble any software engineer is to test their software on real-world candidate footprints.

I had just finished wiring up Breadwinner’s ingestion engine. The promise was straightforward: pass an online footprint, extract verified career accomplishments, and populate an immutable local vault called the Pantry.

Classic developer bias immediately took over: I built the ingestor around my own setup. I maintain this website with a technical blog and an /about page, so I casually assumed every engineer deploys a neatly structured personal portfolio.

In reality? Most people don’t have personal websites (Source: Trust me bro, but also look at your own peer group). Their actual career lives in two places: their LinkedIn profile and their GitHub repositories.

When our first closed-beta tester (shoutout to Umair Mangera, @UmairM13, who also opened our very first open-source PR) tried onboarding, he did what any real candidate would do:

  1. He pasted his LinkedIn URL.
  2. When that felt slow, he pasted his GitHub profile.

Neither worked. Both failed in ways that exposed the fundamental delusion of “just scrape the web with an LLM.”

┌──────────────────────────────────────┐
│       WHAT BROKE ON REAL INPUTS      │
├──────────────────────────────────────┤
│ 01. LinkedIn Authwall Trap           │
│  In:   linkedin.com/in/*             │
│  Fail: HTTP 999 Cookie Gate          │
│  Leak: 400-word GDPR policy in CV    │
│                                      │
│ 02. GitHub "Projects" Blindspot      │
│  In:   github.com/UmairM13           │
│  Fail: Crawled ?tab=projects         │
│  Leak: Empty Kanban board; 0 repos   │
│                                      │
│ 03. Scheme-less Domain Crash         │
│  In:   faizpalwala.tech              │
│  Fail: Missing "https://" scheme     │
│  Leak: 16-char URL as autobiography  │
└──────────────────────────────────────┘

Bug 1: The LinkedIn Authwall Trap (And Avoiding Account Bans)

When you make an unauthenticated HTTP GET request to linkedin.com/in/username, LinkedIn doesn’t serve a profile. It returns an HTTP 999 Request Denied or redirects to a cookie banner: “Join LinkedIn to see who you know.”

Our BeautifulSoup parser stripped the scripts, grabbed the <main> tag, and dutifully passed 400 words of GDPR cookie disclosures to the LLM.

At this point, I’m pretty sure Umair would have happily traded for standard LLM hallucinations instead of watching LinkedIn’s Third-Party Data Processing Agreement bleed into his resume.

Why Not Spin Up Headless Playwright and Steal Cookies?

The standard hack across low-tier scrapers is to ask the user for their li_at session cookie or run headless Playwright with user profile data.

Umair came to me wanting to land more recruiter callbacks. Imagine if my response was:

“Great news: I scraped your profile.
Bad news: you don’t have a LinkedIn account anymore.”

Technically, this would have worked: exactly once. At least he won’t be getting any rejection emails anymore.

Getting a tester banned on a platform they need to get hired is the ultimate anti-feature, and the fastest way to gain new opps.

Can’t get rejection emails if LinkedIn bans your account

The Fix: CLI Boundary Interception

Stop trying to hack around login walls with brittle scrapers. Breadwinner now intercepts LinkedIn URLs at the CLI boundary before firing any network requests:

# breadwinner/core/ingestor.py
if "linkedin.com/in/" in clean_source:
    raise IngestionError(
        "LinkedIn blocks unauthenticated requests (HTTP 999).\n"
        "To ingest your career risk-free without account bans:\n"
        "  1. Open your LinkedIn profile -> Click 'More' -> 'Save to PDF'\n"
        "  2. Run: breadwinner ingest ~/Downloads/Profile.pdf"
    )

By routing users to LinkedIn’s official “Save to PDF” or their Data Archive export (Positions.csv), we achieve 100% metric fidelity, zero network overhead, zero ban risk, and zero risk of losing a friend.

Bug 2: The GitHub “Projects” Blindspot

When Umair pasted his GitHub profile, our classifier recognized an HTTP URL and tagged it as a portfolio.

The ingestor ran discover_portfolio_routes() to find subpages matching standard career keywords:

PRIORITY_SUBPAGES = ('about', 'resume', 'experience', 'projects', 'work')

On GitHub, it spotted an anchor tag containing “projects” and crawled https://github.com/UmairM13?tab=projects.

Except on GitHub, “Projects” isn’t a portfolio of software you built. It is GitHub’s built-in Kanban board feature: which for almost every developer I know is completely empty ("You don't have any projects yet"). No shade to the one person on earth who actually uses GitHub Projects for their life admin, but we felt genuinely gaslit by that response.

The ingestor extracted an empty board, completely missed his actual repositories, and ignored years of code.

The Fix: Native GitHub REST API Ingestion

We bypassed DOM scraping entirely for GitHub user URLs. When github.com/{username} is detected, Breadwinner queries GitHub’s public API directly:

  1. Query https://api.github.com/users/{username}/repos?type=public&sort=updated&per_page=100.
  2. Filter out forks (repo['fork'] is False) to focus strictly on original work.
  3. Rank original repos by star count and recency.
  4. Fetch raw README files directly from raw.githubusercontent.com/{username}/{repo}/HEAD/README.md.
  5. Run the payload through our AST entity deduplicator to populate verified project cards.
flowchart TD
    In["github.com/{username}"] --> API["GitHub Public REST API<br/>(/users/{username}/repos)"]
    API --> Filter["Filter Forks & Empty Stubs"]
    Filter --> Rank["Rank by Stars & Recency"]
    Rank --> Raw["Fetch Raw README.md<br/>(raw.githubusercontent.com)"]
    Raw --> AST["AST Entity Clustering & Deduplication"]
    AST --> Pantry["Pantry (profile.json)"]
    
    style In fill:#1e293b,stroke:#475569,stroke-width:2px,color:#fff
    style Raw fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#fff
    style Pantry fill:#022c22,stroke:#059669,stroke-width:3px,color:#fff

Bug 3: The Bare-Domain Crash (faizpalwala.tech)

When I tested my own personal site, I typed faizpalwala.tech without https://.

In the original ingestor:

if source.startswith("http://") or source.startswith("https://"):
    return self._ingest_url(source)
elif Path(source).exists():
    return self._ingest_file(source)
else:
    return self._ingest_raw_text(source)

It didn’t start with http://. It wasn’t a local path on my SSD. So Python silently fell through to _ingest_raw_text(). It literally treated the 16 characters of my URL as candidate autobiography.

Honestly, bugs like this feel like an unavoidable canon event for anyone writing input parsers.

The Fix: Strict Domain Normalization

We implemented regex protocol normalization before route dispatch:

def normalize_source(source: str) -> tuple[str, str]:
    cleaned = source.strip()
    # Detect bare domains (e.g. faizpalwala.tech or domain.com/about)
    domain_pattern = r"^(?:[a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(?:/.*)?$"
    if re.match(domain_pattern, cleaned) and not cleaned.startswith(("http://", "https://")):
        return f"https://{cleaned}", "url"
    if cleaned.startswith(("http://", "https://")):
        return cleaned, "url"
    if Path(cleaned).exists():
        return cleaned, "file"
    return cleaned, "raw_text"

Craftsmanship: Solve for the Specific Case First, Generalize Later

There is a fundamental software craftsmanship lesson here that I have been reflecting on: It is always easier to solve for the concrete specific case first, and extract generality later, than to build a grand generalization upfront and get bogged down by edge cases you never foresaw.

When you start architecting an ingestion engine, the academic instinct is to design an all-encompassing, polymorphic abstraction: take any arbitrary string or URI, strip the markup, and trust an LLM to magically figure out the semantics. It feels clean. It looks beautiful on a diagram.

In production, that abstraction leaks immediately.

The web isn’t an orderly collection of uniform documents; it is a messy, adversarial ecosystem of login walls, rate-limit gates, SPA route boundaries, and bespoke feature tabs.

A generalized scraper fails on LinkedIn because LinkedIn aggressively resists unauthenticated bots. It fails on GitHub because developer work lives in Git trees and README commits, not in static profile HTML.

Attempting to force all these distinct platforms into a single “generic scrape” didn’t make Breadwinner elegant: it made it brittle.

True craftsmanship in developer tooling isn’t about maintaining the purest abstract interface. It is having the humility to solve for the concrete platform quirks first, establish hard invariants on real-world inputs, and only extract a shared abstraction once working code has earned that right.

The Result

With CLI boundary interception for login walls, native GitHub REST ingestion, and regex domain normalization, we re-ran Breadwinner against real-world candidate footprints:

✓ Ingestion complete! 4 roles, 4 projects, 2 institutions, and 100% metric preservation.

Dogfooding hurts because it forces you to face sloppy assumptions in your own code. But resolving those edge cases is the difference between a brittle demo script and software that actually works.

Breadwinner is now public on GitHub as v0.9.0-beta.