---
{"id": "111774068472243697", "in_reply_to_id": null, "in_reply_to_account_id": null, "sensitive": false, "spoiler_text": "", "visibility": "public", "language": "en", "replies_count": 1, "reblogs_count": 0, "favourites_count": 0, "edited_at": null, "reblog": null, "application": null, "account": {"id": "108219415927856966", "username": "brozek", "acct": "brozek", "display_name": "Brandon Rozek", "url": "https://fosstodon.org/@brozek", "uri": "https://fosstodon.org/users/brozek", "avatar": "https://cdn.fosstodon.org/accounts/avatars/108/219/415/927/856/966/original/bae9f46f23936e79.jpg", "avatar_static": "https://cdn.fosstodon.org/accounts/avatars/108/219/415/927/856/966/original/bae9f46f23936e79.jpg", "header": "https://fosstodon.org/headers/original/missing.png", "header_static": "https://fosstodon.org/headers/original/missing.png", "noindex": true, "roles": []}, "media_attachments": [], "mentions": [], "tags": [], "emojis": [], "card": {"url": "https://ognjen.io/using-robots-txt-to-discover-hidden-content/", "title": "Using robots.txt to discover hidden content", "description": "robots.txt might prevent search engines from indexing content, but it can expose it to non-robots", "language": "en", "type": "link", "author_name": "", "author_url": "", "provider_name": "Ognjen Regoje \u2022 ognjen.io", "provider_url": "", "html": "", "width": 600, "height": 315, "image": "https://cdn.fosstodon.org/cache/preview_cards/images/028/008/248/original/3ec586336edf8355.png", "image_description": "", "embed_url": "", "blurhash": "UGRysg9Ff6Rj00azxut7D%RjaeWBM{t7ayWB", "published_at": null}, "poll": null, "syndication": "https://fosstodon.org/@brozek/111774068472243697", "date": "2024-01-18T00:15:33.476Z"}
---
{{< unsafe >}}
<p>I&#39;m having fun distracting myself with the robots.txt of major websites thanks to Oggy.</p><p><a href="https://ognjen.io/using-robots-txt-to-discover-hidden-content/" target="_blank" rel="nofollow noopener noreferrer" translate="no"><span class="invisible">https://</span><span class="ellipsis">ognjen.io/using-robots-txt-to-</span><span class="invisible">discover-hidden-content/</span></a></p><p>Like why does the verge not want to index the iPhone 15 event? 🤔</p>
{{< /unsafe >}}