⚡ 25% OFF Deal — Ends in 02:59:45✦💸 Was $65 → Now Only $49✦🚀 120+ Claude Code Specialists — One-Time Payment✦⚡ 25% OFF Deal — Ends in 02:59:45✦💸 Was $65 → Now Only $49✦🚀 120+ Claude Code Specialists — One-Time Payment✦⚡ 25% OFF Deal — Ends in 02:59:45✦💸 Was $65 → Now Only $49✦🚀 120+ Claude Code Specialists — One-Time Payment✦⚡ 25% OFF Deal — Ends in 02:59:45✦💸 Was $65 → Now Only $49✦🚀 120+ Claude Code Specialists — One-Time Payment✦
⚡ 25% OFF⏱️ 02:59:45
Free Tool

Robots.txt & Sitemap Configurator

Create search engine crawl directives for your site. Generate both a standard robots.txt file and a Next.js App Router robots.ts handler — ready to copy with one click.

Enter paths you want search engines to skip. Common examples: /api/, /admin/, /dashboard/, /_next/

Next.js (src/app/robots.ts)
import { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  const baseUrl = "https://claudeagents.co";

  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: ["/api/", "/admin/"],
    },
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}
Static (public/robots.txt)
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://claudeagents.co/sitemap.xml
Automate your entire SEO workflow with specialized agents

120+ agents handle schema generation, audits, and code review directly in your terminal.

All Tools →

Frequently Asked Questions

Common questions about robots.txt, crawl directives, and sitemap configuration.