The Obama administration is backing technology to help make the process of governing more open and transparent. ... The robots.txt file the Bush administration set up for Whitehouse.gov ran to almost 2377 lines and thereby stopped search engines logging a lot of the data found on the site. On the first day of the Barack administration the robots.txt file shrunk to two lines allowing, for the moment, search sites to index everything it contains.
Barack Obama White House robots.txt censorship