#folders to be restricted from web crawlers user-agent: * disallow: /old disallow: /old2 disallow: /old3 disallow: /millmanstudios disallow: /vault disallow: /dev disallow: /wordpress #This tells Google's Mediabot to index all of my pages for the Adsense program User-agent: Mediapartners-Google* Disallow: