Stagehand MCP agent

A web scraping agent that uses the Stagehand MCP server to automate browser interactions and create a structured content digest from Hacker News.

Key Features

  • Safe Navigation: Proper initialization sequence prevents common browser automation errors
  • Structured Data Extraction: Methodical approach to extracting and organizing web content
  • Flexible Output: Creates well-structured digests with headlines, summaries, and insights

Prerequisites

Before running this example, you'll need:

  • Browserbase Account: Get API credentials from Browserbase
  • OpenAI API Key: Get an API Key from OpenAI

Setup Instructions

1. Clone and Build Stagehand MCP Server

1git clone https://github.com/browserbase/mcp-server-browserbase
2
3# Navigate to the stagehand directory
4cd mcp-server-browserbase/stagehand
5
6# Install dependencies and build
7npm install
8npm run build

2. Install Python Dependencies

1uv pip install kern-ai mcp openai

3. Set Environment Variables

1export BROWSERBASE_API_KEY=your_browserbase_api_key
2export BROWSERBASE_PROJECT_ID=your_browserbase_project_id
3export OPENAI_API_KEY=your_openai_api_key

Code Example

1import asyncio
2from os import environ
3from textwrap import dedent
4
5from kern.agent import Agent
6from kern.models.openai import OpenAIResponses
7from kern.tools.mcp import MCPTools
8from mcp import StdioServerParameters
9
10
11async def run_agent(message: str) -> None:
12 server_params = StdioServerParameters(
13 command="node",
14 # Update this path to the location where you cloned the repository
15 args=["mcp-server-browserbase/stagehand/dist/index.js"],
16 env=environ.copy(),
17 )
18
19 async with MCPTools(server_params=server_params, timeout_seconds=60) as mcp_tools:
20 agent = Agent(
21 model=OpenAIResponses(id="gpt-5.2"),
22 tools=[mcp_tools],
23 instructions=dedent("""\
24 You are a web scraping assistant that creates concise reader's digests from Hacker News.
25
26 CRITICAL INITIALIZATION RULES - FOLLOW EXACTLY:
27 1. NEVER use screenshot tool until AFTER successful navigation
28 2. ALWAYS start with stagehand_navigate first
29 3. Wait for navigation success message before any other actions
30 4. If you see initialization errors, restart with navigation only
31 5. Use stagehand_observe and stagehand_extract to explore pages safely
32
33 Available tools and safe usage order:
34 - stagehand_navigate: Use FIRST to initialize browser
35 - stagehand_extract: Use to extract structured data from pages
36 - stagehand_observe: Use to find elements and understand page structure
37 - stagehand_act: Use to click links and navigate to comments
38 - screenshot: Use ONLY after navigation succeeds and page loads
39
40 Your goal is to create a comprehensive but concise digest that includes:
41 - Top headlines with brief summaries
42 - Key themes and trends
43 - Notable comments and insights
44 - Overall tech news landscape overview
45
46 Be methodical, extract structured data, and provide valuable insights.
47 """),
48 markdown=True,
49 )
50 await agent.aprint_response(message, stream=True)
51
52
53if __name__ == "__main__":
54 asyncio.run(
55 run_agent(
56 "Create a comprehensive Hacker News Reader's Digest from https://news.ycombinator.com"
57 )
58 )

Available Tools

The Stagehand MCP server provides several tools for web automation:

ToolPurposeUsage Notes
stagehand_navigateNavigate to web pagesUse first for initialization
stagehand_extractExtract structured dataSafe for content extraction
stagehand_observeFind elements and understand page structureGood for exploration
stagehand_actInteract with page elementsClick, type, scroll actions
screenshotTake screenshotsUse only after navigation succeeds