# Insightful-Pipe/web-crawler-mcp-server

Source: https://glimind.com/server/github%2Finsightful-pipe%2Fweb-crawler-mcp-server
Registry id: `github/insightful-pipe/web-crawler-mcp-server`
Status: unknown

## Tools (33)

- `ai-crawler-checker`: Check permissions for AI training bots and crawlers
- `amp-checker`: Verify AMP implementation quality and discoverability
- `brand-extractor`: Extract brand design details from a website (colors, fonts, buttons, logos)
- `canonical-checker`: Validate canonical URLs and conflicting hints
- `character-encoding-checker`: Validate declared charset and encoding consistency
- `content-optimizer-checker`: Analyze on-page content quality and keyword density
- `content-quality-checker`: Score readability, word count, and duplication risk
- `deprecated-html-checker`: Detect obsolete tags or attributes still in use
- `favicon-checker`: Confirm favicon/icon discovery across devices
- `headings-checker`: Review heading hierarchy (H1-H6) for structure gaps
- `hreflang-checker`: Review international hreflang annotations
- `image-optimization-checker`: Check alt text, formats, and compression levels
- `indexability-checker`: Review robots/meta directives affecting indexation
- `internal-linking-checker`: Map internal links and surface opportunities
- `language-checker`: Verify html[lang] attributes and language targeting
- `link-analysis-checker`: Detect broken or low-quality links on page
- `link-extractor`: Extract all links from a page and categorize them (internal and external)
- `llms-txt-checker`: Parse llms.txt instructions for AI crawlers
- `meta-tags-checker`: Inspect meta title/description tags across the page
- `mobile-responsive-checker`: Audit responsive layout rules and breakpoints
- `mobile-viewport-checker`: Ensure viewport meta tags and responsive meta configs are in place
- `page-load-checker`: Measure page weight and render-blocking assets
- `page-speed-checker`: Measure load times and highlight slow resources
- `robots-txt-checker`: Parse robots.txt directives and highlight blocking issues
- `robots-txt-generator`: Generate robots.txt content with optional AI bot restrictions
- `security-headers-checker`: Check CSP/HSTS and other security headers
- `seo-audit-checker`: Comprehensive SEO audit covering technical and content issues
- `sitemap-generator`: Crawl the target domain and produce an XML sitemap
- `social-media-checker`: Validate Open Graph and Twitter Card metadata
- `ssl-certificate-checker`: Inspect certificate chain, expiry, and HTTPS enforcement
- `status-code-checker`: Trace HTTP status/redirect chains for a URL
- `structured-data-checker`: Validate Schema.org markup and structured data coverage
- `url-structure-checker`: Evaluate keyword usage and cleanliness of URLs

Measured by Glimind (https://glimind.com): MCP handshake + tool list only, no tool is ever called. CC BY 4.0.
