Skip to main content

2 posts tagged with "til"

View All Tags

· 3 min read
Luis Gomez

Web scraping has always been fascinating to me – the thrill of harvesting forbidden fruit.

Today I learned just how much things have advanced.

The state of the art of web scraping in 2026:​

Offense​

Tools​

  • Camoufox
  • Residential proxies

Secret Sauce​

  • Proxy rotation.
    • IP flagged as a bot? Try another.
  • Session warming - (the craziest new concept to me that I actually saw make a difference)
    • Search your target website in google, bing, or duckduckgo. Browse around, look normal, smile and wave, etcetera, to build "reputation" for Akamai-type detectors. Once you've done this for about a minute, Akamai seems to let their guard down a bit, and you can scrape to your hearts content, as long as your bot's movements aren't too erratic.
    • Learned while searching through X for clues about how pro scrapers do what they do. Src: https://x.com/mathieulevrai1/status/2054543662021820683?s=20

Defense​

Tools (the web scraper's "final bosses")​

  • Akamai
    • Behavioral tracking. Essentially, inject a script into the browser that watches your actions. Mouse movements, keystroke patterns, and browsing history are all fair game. If any of these are suss, you're flagged and either blocked or served fake data.
  • Cloudflare
    • Fingerprinting. Looks for fishy things like mismatch between the actual TLS fingerprint of a python requests-issued HTTP request and the fingerprint you'd expect from the request's User-Agent.

For more specifics on the techniques each company uses, this AI chat is fairly comprehensive.

TLDR; what works?​

  • Camoufox (thanks to it's advanced fingerprint injection), and
    • rendering in a real display (even XVFB works and has first-party support)
    • a residential proxy pool,
    • and "session warming", described above, pretty reliably bypasses Cloudflare and Akamai checks, even from a VPS. For now.

Where do I think this all goes in the future?​

I don't see how this ends besides a GAN-type war between model apps (e.g. Firecrawl), who rely on sophisticated, dubiously legal scraping pipelines, and security providers (Akamai & Cloudflare), who sell millions of enterprise contracts on protecting your data from relentless bot armies.

Unlike most other software battles, I see the odds being in favor of offense. Realistically, bot's will be able to mimic human browsing behavior perfectly, to the point here someone who really wants to protect their data from scraping needs to put the bulk of it behind a paywall or a paid API. Do you see Amazon.com doing that, dropping conversion rates, and evaporating shareholder value? No chance. But some more boutique websites definitely could. Even then, there will likely still be profit to be had at an aggregation layer which subscribes to expensive data behind paywalls, builds value on top, and captures value internally or exposes a new, higher-level platform which charges it's own spread.

For these reasons, I'm glad I upped my scraping game. It will be a valuable skill for a long time to come.

· 3 min read
Luis Gomez

How to effectively work with Jupyter Notebooks using LLMS​

I discovered some extremely effective ways to work with Jupyter Notebooks with LLM assistance this week.

Things I found massively helpful:

  1. Cell tagging
  2. Agentic cell execution

1. Cell Tagging​

Every Jupyter Notebook is just a JSON file with a specific structure. Each cell in a Jupyter Notebook is just an entry in the cells list. One of the properties a cell has is metadata. Jupyter Notebook clients, like the built-in notebook viewer in VSCode / Cursor, have a feature which leverage this metadata property to add tags.

In VSCode, we can do this by opening the command pallete and running the "Add Cell Tag" command.

The JSON for a cell with source code: print("hello world") tagged with example-tag looks like this:

{
 "cell_type": "code",
 "execution_count": null,
 "id": "16bb7324",
 "metadata": {
  "tags": [
   "example-tag"
  ]
 },
 "outputs": [],
 "source": [
  "print(\"hello world\")"
 ]
}

Using tags allows us to reference a specific cell very easily, in a way both I and my LLM understand.

For example I can prompt my LLM with:

In @notebooks/test.ipynb, in the cell tagged example-tag, change the print statement to all caps

and it would very easily make the change without reading irrelevant cells and bloating context.

In this case, I could just as easily reference the cell by it's index like:

In @notebooks/test.ipynb, In the first cell, change the print statement to all caps

but as a notebook grows to dozens of cells, this becomes untenable.

Using cell tags scales.

2. Agentic Cell Execution​

Here's where my mind was really blown – running cells in agentic loops.

Cell execution with nbconvert​

Say I wanted my LLM not only to make the change to print "hello world" in all caps, I want it to verify that after making the change and running the cell, the output of that cell indeed prints "HELLO WORLD".

Again referencing the cell tag, I can ask my LLM:

"In @notebooks/test.ipynb in the cell tagged example-tag, change the print statement to all caps, run the cell, and verify that the printed text is in all caps"

In this case, your agent should use a tool maintained by the Jupyter Development Team called nbconvert, which, among other things, enables Jupyter notebook execution from the command line.

For our test.ipynb notebook, the command will look like:

jupyter nbconvert --to notebook --execute test.ipynb

This will populate the value of the "outputs" key of your cells. The agent can then look up the cell in question by its tag, read the "outputs" value, and verify the output using its great natural language ability or by writing an ad-hoc script.

Example agent session​

See an example session here walking through what was described above. The agent makes a code change by modifying the "source" value of a tagged cell, uses jupyter nbconvert to execute the notebook, and reads the "outputs" value of the cell to check that our all caps string was printed to stdout.