Does your host or website developer do these things?
Developing a new web site — or re-developing an old one, for that matter — typically involves consulting with the client to determine the site’s target audience and primary objective, creating an attractive and functional design, turning the design into properly coded, valid html, building out the pages of content, and writing the server-side programming to perform whatever dynamic features are needed. But there are several steps that are frequently overlooked before a site “goes live.”
A custom “404 not found” page should be created. At minimum, it should incorporate the site’s overall design and navigation links, and might also include a search form, links to the most popular sections of the site, and/or a way to contact the site owner or webmaster for assistance. And make sure that requests for non-existent pages actually get a “404 page not found” server response. If any other server response is returned — particularly a “200 OK” response — the site could easily become persona non grata in the search engines, among other problems.
A robots.txt file should be created to tell the search engine spiders what pages or parts of the site should not be spidered. Even if you want every page to be spidered, a robots.txt file should be placed in the document root of the site, so as to avoid filling up the site’s error logs with “not found” entries for a non-existent robots.txt. This makes it much easier to spot errors resulting from actual bad links when you examine the error logs.
A canonical URL redirect should be implemented to send all site traffic to your desired canonical URL — either www.example.com, or just example.com (without the www). Whichever you prefer, your should make sure that all traffic to the other form is redirected via a proper 301 redirect.
Test all forms and other interactive features. Submit every form. Attempt to submit forms without required information, or with invalid information. Try to break them. Try really really hard to break them. And make sure that whatever is supposed to be done after the form is submitted actually happens. If an e-mail is supposed to be sent to the site owner, test it, and make sure the owner gets that e-mail with all the appropriate information. If there’s a search engine, search for some things. If the site relies on cron jobs, set them up ahead of time, and make sure they’re running as scheduled and performing as expected.
Test all redirects and rewrites. If the site uses Apache’s mod_rewrite module to present search-engine-friendly URLs, test them all, and test the non-search-engine-friendly versions, to make sure that every bit of content can only be reached by one unique URL.
Check subdirectories for directory listings. Make sure that directory indexes are turned off, and/or for any subdirectory that doesn’t have an index page, plop one in there.
Test all internal links and all outgoing links. Make absolutely sure that every single link leads to the right place. We don’t need to stinkin’ dead links!
There’s simply no excuse for ever launching a site without having completed each of these steps.