Palena WebSearch MCPEnterprise web search for AI.

A Go MCP server that searches the open web, strips PII, screens prompt-injection payloads, and hands back LLM-ready markdown with a full audit trail.

Get Started

Concepts

GitHub

🔎

Metasearch, not scraping-as-a-service

Queries a self-hosted SearXNG instance you control. No third-party API keys, no rate-limited search vendors, no unexpected data exfiltration.

How the pipeline works

🕸️

Tiered content extraction

Plain HTTP + readability first, headless Chromium only when needed, stealth browser with proxy rotation as the last resort. Pay for complexity only when content demands it.

L0 / L1 / L2 extraction

🛡️

PII redaction via Presidio

Microsoft Presidio analyzes every scraped document. Choose between audit, redact, and block modes — enforced per-request, logged per-finding.

PII & compliance

🧱

Prompt-injection defense

A DeBERTa-v3 classifier sidecar screens scraped content before it reaches the model. Annotate suspicious chunks, block them outright, or just audit — your call.

Injection defense

🎯

Pluggable reranker

Swap between a KServe cross-encoder (mxbai-rerank), a lightweight FlashRank ONNX sidecar, an LLM-as-reranker, or no reranker at all. Config-driven, zero code changes.

Reranking options

🧾

Full provenance trail

Every scrape emits a three-stage SHA-256 hash chain. Every PII and injection finding is logged. Auditors get deterministic replay, not hand-wavy summaries.

Provenance details

📡

MCP-native, SSE + Streamable HTTP

Drops into Claude Desktop, Cursor, Windsurf, LibreChat, or any MCP client. Also exposes a standalone REST endpoint if you want direct access.

Client integrations

📊

OpenTelemetry out of the box

Structured logs, Prometheus metrics, and OTLP traces cover every stage. Export to ClickHouse, Jaeger, Grafana — whatever your SRE team already runs.

Observability

🐳

Docker Compose to Kubernetes

A single compose file to run the whole stack locally. A Helm chart for OpenShift and vanilla Kubernetes when you're ready to put it in front of real traffic.

Deployment guide

Why Palena exists

Foundation models cannot safely ingest raw third-party HTML. It carries PII, prompt-injection payloads, adversarial instructions, tracking artifacts, and noise that blows up token budgets.

Palena is the boundary layer — the name means boundary or limit in Hawaiian. It sits between your agent and the open web, enforces the policies your compliance team actually cares about, and returns clean, hashed, audited markdown your model can trust.

What you get

A single MCP tool (web_search) that orchestrates six stages: search → scrape → PII detection → prompt-injection screening → rerank → markdown formatting.
Sidecar architecture — SearXNG, Presidio, Playwright, the injection classifier, and the reranker are all separate containers. Swap them, scale them, replace them independently.
Config-driven behavior — PII mode, injection mode, reranker backend, scraper tiers, domain allow/blocklists, rate limits — all declared in palena.yaml, not burned into code.
Compliance-first defaults — robots.txt enforcement, per-domain rate limits, content hash chains, and audit records that survive a regulator's request.

Who it's for

Fintech, healthtech, govtech, and any team whose legal team has concerns about raw web content reaching an LLM prompt. If you've been told "we can't ship unfiltered HTML into a model," Palena is the filter.

Start with Getting Started to bring up the stack, then read Concepts for the full pipeline picture.