Stagehand MCP agent
A web scraping agent that uses the Stagehand MCP server to automate browser interactions and create a structured content digest from Hacker News.
Key Features
- Safe Navigation: Proper initialization sequence prevents common browser automation errors
- Structured Data Extraction: Methodical approach to extracting and organizing web content
- Flexible Output: Creates well-structured digests with headlines, summaries, and insights
Prerequisites
Before running this example, you'll need:
- Browserbase Account: Get API credentials from Browserbase
- OpenAI API Key: Get an API Key from OpenAI
Setup Instructions
1. Clone and Build Stagehand MCP Server
1git clone https://github.com/browserbase/mcp-server-browserbase23# Navigate to the stagehand directory4cd mcp-server-browserbase/stagehand56# Install dependencies and build7npm install8npm run build2. Install Python Dependencies
1uv pip install kern-ai mcp openai3. Set Environment Variables
1export BROWSERBASE_API_KEY=your_browserbase_api_key2export BROWSERBASE_PROJECT_ID=your_browserbase_project_id3export OPENAI_API_KEY=your_openai_api_keyCode Example
1import asyncio2from os import environ3from textwrap import dedent45from kern.agent import Agent6from kern.models.openai import OpenAIResponses7from kern.tools.mcp import MCPTools8from mcp import StdioServerParameters91011async def run_agent(message: str) -> None:12 server_params = StdioServerParameters(13 command="node",14 # Update this path to the location where you cloned the repository15 args=["mcp-server-browserbase/stagehand/dist/index.js"],16 env=environ.copy(),17 )1819 async with MCPTools(server_params=server_params, timeout_seconds=60) as mcp_tools:20 agent = Agent(21 model=OpenAIResponses(id="gpt-5.2"),22 tools=[mcp_tools],23 instructions=dedent("""\24 You are a web scraping assistant that creates concise reader's digests from Hacker News.2526 CRITICAL INITIALIZATION RULES - FOLLOW EXACTLY:27 1. NEVER use screenshot tool until AFTER successful navigation28 2. ALWAYS start with stagehand_navigate first29 3. Wait for navigation success message before any other actions30 4. If you see initialization errors, restart with navigation only31 5. Use stagehand_observe and stagehand_extract to explore pages safely3233 Available tools and safe usage order:34 - stagehand_navigate: Use FIRST to initialize browser35 - stagehand_extract: Use to extract structured data from pages36 - stagehand_observe: Use to find elements and understand page structure37 - stagehand_act: Use to click links and navigate to comments38 - screenshot: Use ONLY after navigation succeeds and page loads3940 Your goal is to create a comprehensive but concise digest that includes:41 - Top headlines with brief summaries42 - Key themes and trends43 - Notable comments and insights44 - Overall tech news landscape overview4546 Be methodical, extract structured data, and provide valuable insights.47 """),48 markdown=True,49 )50 await agent.aprint_response(message, stream=True)515253if __name__ == "__main__":54 asyncio.run(55 run_agent(56 "Create a comprehensive Hacker News Reader's Digest from https://news.ycombinator.com"57 )58 )Available Tools
The Stagehand MCP server provides several tools for web automation:
| Tool | Purpose | Usage Notes |
|---|---|---|
stagehand_navigate | Navigate to web pages | Use first for initialization |
stagehand_extract | Extract structured data | Safe for content extraction |
stagehand_observe | Find elements and understand page structure | Good for exploration |
stagehand_act | Interact with page elements | Click, type, scroll actions |
screenshot | Take screenshots | Use only after navigation succeeds |