phpBB and sessions ID, Search Engine optimization technique:Technology & Computers

During working on this forum and few others i came to understand different techniques to make phpBB search engine friendly so I decided to summ up all .
Here I have collected all what i have found...

Right in the begining there are two mods (php+google) and (google-enhancement-mode) The first one is used to remove the SIDs for google and google like search engines and the second one is to give google a single session.
I found that both the mods work fine untill the version 2.0.3 of phpBB. and the later versions reaquires some changes. So people later to phpBB
2.0.3 should use the following

Google Single Session mod #1

Code:




includes/sessions.php 


   global $SID; 

if ( !empty($SID) && !preg_match('#sid=#', $url) ) 


   global $SID, $HTTP_SERVER_VARS; 

   if ( !empty($SID) && !preg_match('sid=', $url) && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Googlebot') && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'slurp@inktomi.com'))

Also there are several other version of this mod.. but the should be this untill i m writhing this.. I have got this conclusion from the feedback of various other users.
Now in addition to that mod...people should have robots.txt in ther root directory.. (here assumes that u have installed ur forum in root otherwise change the path respective to ur forums path)..

Code:

User-agent: * 
Disallow: /admin/ 
Disallow: /attach_mod/ 
Disallow: /db/ 
Disallow: /files/ 
Disallow: /images/ 
Disallow: /includes/ 
Disallow: /language/ 
Disallow: /mycalendar_mod/ 
Disallow: /spelling/ 
Disallow: /templates/ 
Disallow: /common.php 
Disallow: /config.php 
Disallow: /glance_config.php 
Disallow: /groupcp.php 
Disallow: /memberlist.php 
Disallow: /mini_cal.php 
Disallow: /modcp.php 
Disallow: /mycalendar.php 
Disallow: /news_insert.php 
Disallow: /posting.php 
Disallow: /printview.php 
Disallow: /privmsg.php 
Disallow: /profile.php 
Disallow: /ranks.php 
Disallow: /search.php 
Disallow: /statistics.php 
Disallow: /tellafriend.php 
Disallow: /viewonline.php

The robots.txt don't let google to spider the unwanted posts..
-------------------------------------------------------------------
Also there was also a another approach to get rid of SIds for guest and search engines.. That is another mod. But this mods don't work if u allow annonymous user on to post on ur site...
SID removal for all #2

Code:

# 


includes/sessions.php 




$SID = 'sid=' . $session_id; 




if ( $userdata['session_user_id'] != ANONYMOUS ){ 
   $SID = 'sid=' . $session_id; 
} else { 
   $SID = ''; 
}

Here also inclusion of the above provided robots.txt is appreciated.

-------------------------------
Below is the mod.. can be used with Mod #1 to get indexed in google faster..
It requires mod_rewrite enabled.. as directed below. (This mod is not in standard form but u can understand it.

Code:

Requirements: you need mod_rewrite enabled as well as ability to use .htaccess or modify apache config files. 

Implementation: 

Step 1. In /includes/page_header.php before 

Code: 
// 
// Generate logged in/logged out status 
// 


add this code (make sure there are no space breaks at line ends after you paste): 

Code: 
ob_start(); 
function replace_for_mod_rewrite(&$s) 
{ 
$urlin = 
array( 
"'(?<!/)viewforum.php\?f=([0-9]*)&topicdays=([0-9]*)&start=([0-9]*)'", 
"'(?<!/)viewforum.php\?f=([0-9]*)&mark=topics'", 
"'(?<!/)viewforum.php\?f=([0-9]*)'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)&view=previous'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)&view=next'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)&postdays=([0-9]*)&postorder=([a-zA-Z]*)&start=([0-9]*)'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)&start=([0-9]*)&postdays=([0-9]*)&postorder=([a-zA-Z]*)&highlight=([a-zA-Z0-9]*)'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)&start=([0-9]*)'", 
"'(?<!/)viewtopic.php\?t=([0-9]*)'", 
"'(?<!/)viewtopic.php&p=([0-9]*)'", 
"'(?<!/)viewtopic.php\?p=([0-9]*)'", 
); 
$urlout = array( 
"viewforum\\1-\\2-\\3.html", 
"forum\\1.html", 
"forum\\1.html", 
"ptopic\\1.html", 
"ntopic\\1.html", 
"ftopic\\1-\\2-\\3-\\4.html", 
"ftopic\\1.html", 
"ftopic\\1-\\2.html", 
"ftopic\\1.html", 
"sutra\\1.html", 
"sutra\\1.html", 
); 
$s = preg_replace($urlin, $urlout, $s); 
return $s; 
} 


Step 2. In /includes/page_tail.php after 

Code: 
$db->sql_close(); 


add this: 

Code: 
$contents = ob_get_contents(); 
ob_end_clean(); 
echo replace_for_mod_rewrite($contents); 
global $dbg_starttime; 
 


in the same file after 

Code: 
ob_end_clean(); 


add this: 

Code: 
echo replace_for_mod_rewrite($contents); 
global $dbg_starttime; 


Step 3. In your .htaccess file (if you don't have one create it; should be located in you forum root directory) paste these lines: 

Code: 
RewriteEngine On 
RewriteRule ^forums.* index.php 
RewriteRule ^forum([0-9]*).* viewforum.php?f=$1&mark=topic 
RewriteRule ^viewforum([0-9]*)-([0-9]*)-([0-9]*).* viewforum.php?f=$1&topicdays=$2&start=$3 
RewriteRule ^forum([0-9]*).* viewforum.php?f=$1 
RewriteRule ^ptopic([0-9]*).* viewtopic.php?t=$1&view=previous 
RewriteRule ^ntopic([0-9]*).* viewtopic.php?t=$1&view=next 
RewriteRule ^ftopic([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 
RewriteRule ^ftopic([0-9]*)-([0-9]*).* viewtopic.php?t=$1&start=$2 
RewriteRule ^ftopic([0-9]*).* viewtopic.php?t=$1 
RewriteRule ^ftopic([0-9]*).html viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 
RewriteRule ^sutra([0-9]*).* viewtopic.php?p=$1 


Note: sometimes this .htaccess code won't work properly if the forum is on a subdomain (was the case with me), you may need to try this variation: 

Code: 
RewriteEngine On 
RewriteRule ^forums.* /index.php 
RewriteRule ^forum([0-9]*).* /viewforum.php?f=$1&mark=topic 
RewriteRule ^viewforum([0-9]*)-([0-9]*)-([0-9]*).* /viewforum.php?f=$1&topicdays=$2&start=$3 
RewriteRule ^forum([0-9]*).* /viewforum.php?f=$1 
RewriteRule ^ptopic([0-9]*).* /viewtopic.php?t=$1&view=previous 
RewriteRule ^ntopic([0-9]*).* /viewtopic.php?t=$1&view=next 
RewriteRule ^ftopic([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* /viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 
RewriteRule ^ftopic([0-9]*)-([0-9]*).* /viewtopic.php?t=$1&start=$2 
RewriteRule ^ftopic([0-9]*).* /viewtopic.php?t=$1 
RewriteRule ^ftopic([0-9]*).html /viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 
RewriteRule ^sutra([0-9]*).* /viewtopic.php?p=$1 


Step 4. Very important! In your robots.txt file (goes at the *site* root) add these lines: 

Code: 
Disallow: /your-forum-folder/sutra*.html$ 
Disallow: /your-forum-folder/ptopic*.html$ 
Disallow: /your-forum-folder/ntopic*.html$ 
Disallow: /your-forum-folder/ftopic*asc*.html$ 


(This is required to avoid feeding duplicate content to Google) 

Step 5. Apply Google Mod #1 described at this page.

Then there was a suggestion to make a sitemap page having static url to each topic in ur forum generated dynamically. This will allow google to get ur all topics indexed without Sids. But some suggest it is not a good idea for big forums because google don't like more than 200 links on a page. Well the sitemap page u need is below..

Code:


<?php 
/*************************************************************************** 
* Save this file as&#58;   site_map.php &#40;or anything you like&#41; 
* Version&#58;      Friday, Oct 4, 2002 
* Email&#58;      angus@phphacks.com 
* Purpose of hack&#58;   Basically generates a list of topics and 
*      displays them with link to the topic. Goal 
*      is to provide search engines like Google 
*      with a static page of links to dynamic pages 
*      You should link to this page from your sites 
*      home page somewhere. 
* Demo&#58;      http&#58;//www.aussiecelebs.com/forums/site_map.php 
* Tested on&#58;   phpBB 2.01, 2.02 
* 
***************************************************************************/ 

/*************************************************************************** 
* 
* This program is free software; you can redistribute it and/or modify 
* it under the terms of the GNU General Public License as published by 
* the Free Software Foundation; either version 2 of the License, or 
* &#40;at your option&#41; any later version. 
* 
***************************************************************************/ 

echo " 

<!-- Start header here. Make sure you dont use any \" s --> 

<html> 
<head> 
<meta http-equiv='Content-Language' content='en-au'> 
<title>Enter Your Title Here</title> 
<meta name='keywords' content='keyword phrase, keyword phrase'> 
<meta name='description' content='Site description including reference to keyword prhase'> 
<base target='_top'> 
</head> 
<body bgcolor='#FFFFFF'> 
<h1>Site Map</h1> 
<!-- End Header --> 

"; 

define&#40;'IN_PHPBB', true&#41;; 
$phpbb_root_path = './'; 
include&#40;$phpbb_root_path . 'extension.inc'&#41;; 
include&#40;$phpbb_root_path . 'common.'.$phpEx&#41;; 

echo "<h2>All Categories</h2>"; 

$result = mysql_query&#40;"SELECT cat_title, cat_id FROM phpbb_categories ORDER BY cat_title"&#41;; 
while&#40;$row=  mysql_fetch_assoc&#40;$result&#41;&#41; 
&#123; 
      echo "<a href='index.php?c=".$row["cat_id"&#93;."'>".$row["cat_title"&#93;."</a><br>"; 

&#125; 

echo "<h2>All Forums</h2>"; 

$result = mysql_query&#40;"SELECT forum_id, forum_name, forum_desc FROM phpbb_forums ORDER BY forum_name"&#41;; 
while&#40;$row=  mysql_fetch_assoc&#40;$result&#41;&#41; 
&#123; 
      echo "<a href='viewforum.php?f=".$row["forum_id"&#93;."'>".$row["forum_name"&#93;."</a>"." - ".$row["forum_desc"&#93;."<br>"; 

&#125; 

echo "<h2>All Topics</h2>"; 

$result = mysql_query&#40;"SELECT topic_title, topic_id FROM phpbb_topics ORDER BY topic_title"&#41;; 
while&#40;$row=  mysql_fetch_assoc&#40;$result&#41;&#41; 
&#123; 
      echo "<a href='viewtopic.php?t=".$row["topic_id"&#93;."'>".$row["topic_title"&#93;."</a><br>"; 

&#125; 

echo " 

<!-- Start footer here. --> 

</body> 
</html> 

<!-- End footer --> 

"; 

?>

Place this page inside ur forum directory and give a link to ur site map in ur mainpage.

In addition to use that .. u can also use dynamic metatage generator by doing the following..

Code:


 includes/page_header.php 
find 
Code: 
define('HEADER_INC', TRUE); 

after, add 
Code: 

// MOD TOPIC META TAGS BEGIN 
// add meta tags - we only want to do this is specifaclly asked 
if( isset($HTTP_GET_VARS['with_meta']) || isset($HTTP_POST_VARS['with_meta']) ) 
{ 
    if ( isset($HTTP_GET_VARS[POST_TOPIC_URL]) ) 
    { 
       $topic_id = intval($HTTP_GET_VARS[POST_TOPIC_URL]); 
    } 
    else if ( isset($HTTP_GET_VARS['topic']) ) 
    { 
       $topic_id = intval($HTTP_GET_VARS['topic']); 
    } 
    
    if ( $topic_id ) 
    { 
        $sql = "SELECT c.cat_title, f.forum_name, t.topic_title 
                FROM " . TOPICS_TABLE . " t, " . FORUMS_TABLE . " f, " . CATEGORIES_TABLE . " c 
                WHERE f.forum_id = t.forum_id 
                AND c.cat_id = f.cat_id 
                AND t.topic_id = $topic_id"; 
        if( ($result = $db->sql_query($sql)) ) 
        { 
            if ( $meta_row = $db->sql_fetchrow($result) ) 
            { 
                $meta_description = '<meta name=description content="' . $meta_row['cat_title'] . ' :: ' . $meta_row['forum_name'] . ' :: ' . $meta_row['topic_title'] . '">'; 
            } 
        } 
        $sql = "SELECT w.word_text 
                FROM " . TOPICS_TABLE . " t, " . SEARCH_MATCH_TABLE . " m, " . SEARCH_WORD_TABLE . " w 
                WHERE t.topic_first_post_id = m.post_id 
                AND m.word_id = w.word_id 
                AND t.topic_id = $topic_id"; 
        if( ($result = $db->sql_query($sql)) ) 
        { 
            $meta_keywords = ''; 
            while ( $meta_row = $db->sql_fetchrow($result) ) 
            { 
                $meta_keywords .= ($meta_keywords=='') ? $meta_row['word_text'] : ',' . $meta_row['word_text']; 
            } 
            $meta_keywords = '<meta name=keywords content="' . $meta_keywords . '">'; 
        }    
    } 
} 
// MOD TOPIC META TAGS END 
 


find 
Code: 
'PAGE_TITLE' => $page_title, 

after, add 
Code: 

    'META_DESCRIPTION' => $meta_description, 
    'META_KEYWORDS' => $meta_keywords, 
 


in templates/subSilver/overall_header.tpl 
find 
Code: 
<meta http-equiv="Content-Style-Type" content="text/css"> 

after, add 
Code: 

{META_DESCRIPTION} 
{META_KEYWORDS} 
 


then all you need to do in your sitemap page is add... 
Code: 
&with_meta=

This another mod by bradleyk

Code:





includes/sessions.php 


function append_sid($url, $non_html_amp = false) 


function tep_not_null($value) { 
    if (is_array($value)) { 
      if (sizeof($value) > 0) { 
        return true; 
      } else { 
        return false; 
      } 
    } else { 
      if (($value != '') && (strtolower($value) != 'null') &&       (strlen(trim($value)) > 0)){ 
        return true; 
      } else { 
        return false; 
      } 
    } 
} 


   global $SID; 

if ( !empty($SID) && !preg_match('#sid=#', $url) ) 


    $user_agent = strtolower($HTTP_SERVER_VARS['HTTP_USER_AGENT']); 
    $spider_flag = false; 

    if (tep_not_null($user_agent)) { 
      $spiders = file($phpbb_root_path . 'includes/spiders.txt'); 

      for ($i=0, $n=sizeof($spiders); $i<$n; $i++) { 
        if (tep_not_null($spiders[$i])) { 
          if (is_integer(strpos($user_agent, trim($spiders[$i])))) { 
            $spider_flag = true; 
            break; 
          } 
        } 
      } 
    } 


        if ( !empty($SID) && !eregi('sid=', $url) && 
             !($spider_flag)) 


includes/spiders.txt 


almaden.ibm.com 
appie 1.1 
architext 
ask jeeves 
asterias2.0 
augurfind 
baiduspider 
bannana_bot 
bdcindexer 
crawler 
crawler@fast 
docomo 
fast-webcrawler 
fluffy the spider 
frooglebot 
geobot 
googlebot 
gulliver 
henrythemiragorobot 
ia_archiver 
infoseek 
kit_fireball 
lachesis 
lycos_spider 
mantraagent 
mercator 
moget/1.0 
muscatferret 
nationaldirectory-webspider 
naverrobot 
ncsa beta 
netresearchserver 
ng/1.0 
osis-project 
polybot 
pompos 
scooter 
seventwentyfour 
sidewinder 
sleek spider 
slurp/si 
slurp@inktomi.com 
steeler/1.3 
szukacz 
t-h-u-n-d-e-r-s-t-o-n-e 
teoma 
turnitinbot 
ultraseek 
vagabondo 
voilabot 
w3c_validator 
zao/0 
zyborg/1.0

The above mod gives some error using Admin CP but you can fix that by making a file admin/includes/spiders.txt
Also there are few more mods and they are good. but the above i have mention turns out to be the successfull and checked by various users feedback.

Also I am very-very thankful to those who had worked a lot to provide varoious mods in this thread.
Well if you find and any errors in it then plz point out..
Thanx..