{"account":{"acct":"brozek","avatar":"https://cdn.fosstodon.org/accounts/avatars/108/219/415/927/856/966/original/bae9f46f23936e79.jpg","display_name":"Brandon Rozek","header":"https://fosstodon.org/headers/original/missing.png","id":"108219415927856966","uri":"https://fosstodon.org/users/brozek","url":"https://fosstodon.org/@brozek","username":"brozek"},"application":null,"card":{"author_name":"","author_url":"","blurhash":"UGRysg9Ff6Rj00azxut7D%RjaeWBM{t7ayWB","description":"robots.txt might prevent search engines from indexing content, but it can expose it to non-robots","embed_url":"","height":315,"html":"","image":null,"image_description":"","language":"en","provider_name":"Ognjen Regoje • ognjen.io","provider_url":"","published_at":null,"title":"Using robots.txt to discover hidden content","type":"link","url":"https://ognjen.io/using-robots-txt-to-discover-hidden-content/","width":600},"content":"<p>I&#39;m having fun distracting myself with the robots.txt of major websites thanks to Oggy.</p><p><a href=\"https://ognjen.io/using-robots-txt-to-discover-hidden-content/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\" translate=\"no\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">ognjen.io/using-robots-txt-to-</span><span class=\"invisible\">discover-hidden-content/</span></a></p><p>Like why does the verge not want to index the iPhone 15 event? 🤔</p>","date":"2024-01-18T00:15:33.476Z","edited_at":null,"emojis":[],"favourites_count":0,"id":"111774068472243697","in_reply_to_account_id":null,"in_reply_to_id":null,"language":"en","media_attachments":[],"mentions":[],"poll":null,"reblog":null,"reblogs_count":0,"replies_count":1,"sensitive":false,"spoiler_text":"","syndication":"https://fosstodon.org/@brozek/111774068472243697","tags":[],"visibility":"public"}