website-toots/111774068472243697.md

id	in_reply_to_id	in_reply_to_account_id	sensitive	spoiler_text	visibility	language	replies_count	reblogs_count	favourites_count	edited_at	reblog	application	account	media_attachments	mentions	tags	emojis	card	poll	syndication	date
111774068472243697	null	null	false		public	en	1	0	0	null	null	null	id username acct display_name url uri avatar avatar_static header header_static noindex roles 108219415927856966 brozek brozek Brandon Rozek https://fosstodon.org/@brozek https://fosstodon.org/users/brozek https://cdn.fosstodon.org/accounts/avatars/108/219/415/927/856/966/original/bae9f46f23936e79.jpg https://cdn.fosstodon.org/accounts/avatars/108/219/415/927/856/966/original/bae9f46f23936e79.jpg https://fosstodon.org/headers/original/missing.png https://fosstodon.org/headers/original/missing.png true					url title description language type author_name author_url provider_name provider_url html width height image image_description embed_url blurhash published_at https://ognjen.io/using-robots-txt-to-discover-hidden-content/ Using robots.txt to discover hidden content robots.txt might prevent search engines from indexing content, but it can expose it to non-robots en link Ognjen Regoje • ognjen.io 600 315 https://cdn.fosstodon.org/cache/preview_cards/images/028/008/248/original/3ec586336edf8355.png UGRysg9Ff6Rj00azxut7D%RjaeWBM{t7ayWB null	null	https://fosstodon.org/@brozek/111774068472243697	2024-01-18T00:15:33.476Z

{{< unsafe >}}

I'm having fun distracting myself with the robots.txt of major websites thanks to Oggy.

https://ognjen.io/using-robots-txt-to-discover-hidden-content/

Like why does the verge not want to index the iPhone 15 event? 🤔

{{< /unsafe >}}