OpenAI has launched GPTBot

According to OpenAI’s blog post, web pages crawled with GPTBot could potentially be utilized to enhance accuracy and expand the capabilities of upcoming iterations of their AI models. A web crawler (web spider), is a type of bot that indexes website content across the internet. Popular search engines like Google and Bing employ web crawlers to enable websites to appear in search results. OpenAI clarified that their web crawler will gather publicly available data from the world wide web while excluding sources with paywalled content, personally identifiable information, or text that violates their policies.