The robots.txt file is used to communicate to the search engines-Google, Yahoo, Bing, etc. BEFORE they crawl a web page, “Hey, look over here, don’t index me!”.  This may be done for a number of reasons, one of which is to prevent duplicate content or pages that really don’t benefit the end users.
The files may be disallowed on any directory level. If you don’t want your website to appear at all, the “disallow: / ” may be used.

There are a few important factors to know:
1)The robots don’t have to follow these rules as this is a non-standardized protocol.
2)If you are playing a trick on the engine- guess what? The robots.text file is public. Not a good idea.
3)If someone links to a robots.txt file then it may be too late to block the engines. In this case you can use the no index/no-follow tag.

For more information regarding the Robots.txt file visit Robots.org 

Meet Joe Chura

Joe began his career on the Ford assembly line at age 20, reading his college textbooks seconds at a time in between building cars. Over the next decade Joe gained experience at many different levels in the automotive industry, including running Ford’s regional sales team and a stint as General Manager of two dealerships, where he increased internet sales by 300%. Combining his passion for computer programming and innovation, Joe co-founded Launch Digital Marketing (LDM) and Dealer Inspire (DI) to bring new retail technology and better online experiences to both car dealers and shoppers.