Referrer Logs : Use referrer logs to analyze your search engine traffic

Referrer logs provide the most important information you can get for marketing your Web site. These logs record a variety of information about visitors to your Web site. The most important to Web site marketers is information on which search engines were used to find your Web site and which keywords they searched on.

The referrer log also tells you the address of the Web site (other than a search engine) the visitor came from. In all likelihood, the site listed in the referrer log has a link to your Web site on one of the pages indicated in the log. This can give you an idea which links from which kinds of sites are working.

Most Web servers can provide referrer logs, but you may have to ask your Web hosting provider to set this up for you. If they won't, or if they claim they can't, change hosting providers immediately. This information is too valuable to do without.

The referrer log includes information about the type of browser the visitor was using and whether or not they downloaded the whole page or only a part of it (which would suggest that it took too long and they moved on).

Information contained in the referrer log can shed light on how people are searching and finding your Web site. For instance, we were once under the mistaken impression that very few people on the Web searched with multiple keywords. We believed that the vast majority of people surfing the Web searched on one keyword at a time and chose a Web site to visit from those results. To our surprise, after reviewing referrer logs we discovered that most people visiting one of our client's sites were searching on multiple words.

Referrer log data are often retrieved by logging onto the Web hosting provider's server with an FTP client like WS_ FTP_ 95. The log file is usually stored in a file on the hosting provider's server and can be quickly downloaded as a text file to your local hard drive.
There are myriad log analysis programs on the market that will take the volumes of data contained in the log and help you to easily understand it graphically.

However, if you don't have such a tool, you can simply open the log file within a text editor or in Microsoft Word (or other word processor). The log file contains strings of data that at first glance look rather difficult to understand, but if you look closely, there are only about eight entries and they are easy to interpret.

When referrer log information is captured, here's an example of what you will see:
198.178.25.131 --[07/ Jan/ 1998: 14: 09: 45 -0800] "GET /onepixel. gif HTTP/ 1.0" 200 43 "http:// www. preowned. com/" "Mozilla/ 2.0 (compatible; MSIE 3.0; Windows 95)"

Here is what each part of this log file means. Some referrer logs will contain a few more pieces of information, some a few less. These are what I consider the most important entries:

Referrer Log Entry Meaning

198.178.25.131 The remote host name: the IP address that identifies the location of the visitor's Internet Service Provider.

[07/ Jan/ 1998: 14: 09: 45 -0800] The date and time of the request or access of
the Web site.

GET /onepixel. gif The actual request - the name of the first file downloaded by the browser. A search engine's spider will almost always request the "robots. txt" file. You can review the section on robots and spider blocking to learn more about search spider's activities.
200 The status code of the request. (Status code
"200" means a completed request).

43 The number of bytes that were transferred to the browser. In this case, the first graphic downloaded was 43 bytes.

http:// www. preowned. com/ The referrer site, or the previous URL that the visitor came from. This is gold to Web marketers. This tells you which search engine or other Web site sent you this traffic. When this entry contains a search engine name, it will also include the keywords that were searched for. See an example of this below.

"Mozilla/ 2.0 (compatible; MSIE 3.0; Windows 95)" The user agent or browser information - in this case the visitor was using Internet Explorer v3.0. Keep an eye on this entry as it will tell you if visitors to your Web site are using current technology. If, for instance, you learn that a large percentage of your visitors are using older browsers you should make sure that you aren't using frames or tables as some older browsers don't support them.

7 See, this log is not so hard to read. Now, look at another log entry:
208.207.98.111 --[08/ Jan/ 1998: 18: 19: 11 -0800] "GET /home. html HTTP/ 1.0" 200 4723
"http:// www. altavista. digital. com/ cgi-bin/ query? pg= aq& text= yes& d0= 1% 2fjan %2f98& q= ballroom+ dance% 2a +OR+ ballroom dancing% 2a& stq= 30" "Mozilla/ 2.0 (compatible; MSIE 3.0; SK; Windows 95)"

Note that the referring Web site is the AltaVista search engine. Pay special attention to this section:

%2f98& q= ballroom+ dance% 2a +OR+ ballroom dancing% 2a& stq= 30" When you remove the code between the percentage signs you are left with what the visitor queried in the AltaVista search engine: ballroom+ dance OR ballroom dancing

A fast and easy way to search through a large log file for keywords that were searched on in search engines is to use the "find" command in your text editor or word processor and search the document for "cgi" as a keyword. Most of these will be search engines. You can also search on the individual search engine names using the "find" command.

We don't yet recommend a particular log file analysis software program, but you can go to any major search engine and search for the keywords "referrer log software" and see what you get. Hopefully, they are using this book too and have optimized their pages to rank well for such a search.