User-agent: * Disallow: /cgi-bin/ ## An appropriate "robots.txt" for short URLs might include the following:# Disallow: /mw/index.php# Do not use the above robots.txt file if you have not implemented short URLs, # otherwise you will block all pages from being indexed!#Disallow: /jobsearch/catsDisallow: /portal/catsDisallow: /staffingDisallow: /dbbackup