This allows all robots to crawl all files. User-agent: * Disallow: