亚瑟av亚洲精品一区二区,无码国内精品久久人妻

workingbeijing

白手起家

論壇徽章:: 0

電梯直達

1樓 [收藏(0)] [報告]

發(fā)表于 2008-03-03 18:24 |只看該作者 |倒序瀏覽

10可用積分

這是一份開源的代碼,但是似乎有點問題,只能取出英文的關(guān)鍵詞,似乎還是有問題的,YUI
例子里用的YUI,這個我沒有上傳

我想分析和統(tǒng)計整個網(wǎng)站的關(guān)鍵詞和出現(xiàn)的頻率和分數(shù)
請問有沒有好的實現(xiàn)方法

keyworddensity.rar (5.44 KB, 下載次數(shù): 76)

[ 本帖最后由 workingbeijing 于 2008-3-3 18:39 編輯 ]

文庫|博客

使用正則表達式與lex實現(xiàn)詞法分析器
C語言的MIPS匯編實現(xiàn)（四）SWITCH
Requested init /linuxrc failed (error -2).
比較 csv 文件中數(shù)據(jù)差異
LMD ElPack v2019.7新版亮點：Transparent mode全新升級|附下載

workingbeijing

白手起家

論壇徽章:: 0

2樓 [報告]

發(fā)表于 2008-03-04 09:29 |只看該作者

config.ini
<?php
set_time_limit(360);
//define('URL_SEED','http://games.hawkenterprises.org/keyworddensity/test.html');//trailing slash
//define('DOMAIN','hawkenterprises.com');
define('SINGLE_SITE',true);
define('SINGLE_PAGE',true);
define('DROP_COUNT',0); // drop keywords that appear less than 4 times

$exclude_terms_url = array(0=>'javascript:','mailto:','file:','.pdf','.jpg','.gif','.png','.doc','#','.xls','.tar','.gz','feed:');
$dropwords = array(0=>'and','the','is','it','a',' ','an','or','of','on','for','to');
?>

run.php
<?php
function get_document($urltofetch){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urltofetch);

curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$str = curl_exec($ch);
curl_close($ch);
return $str;
}
function parsed_document_links($document_string){
$hrefs = array();
$doc = new DOMDocument();
@$doc->loadHTML($document_string);
$anchor_tags = $doc->getElementsByTagName('a');
foreach($anchor_tags as $anchor){
      $hrefs[] = $anchor->getAttribute('href');
}
return $hrefs;
}
function normalize_links($links_array){
global $exclude_terms_url;
$url_normalized = array();
foreach($links_array as $k=>$v){
      $skip_url = false;
      foreach($exclude_terms_url as $key=>$value){
         if(stristr($v,$value) !== false)
            $skip_url = true;
      }
      if(!$skip_url){
         $parsed_link = @parse_url($v);
         if(SINGLE_SITE){
            if(isset($parsed_link['scheme']) && $parsed_link['scheme'] == 'http'){
                  if(stristr($v,DOMAIN) !== false){
                     $url_normalized[] = $v;
                  }
            }else{
                  $url_normalized[] = URL_SEED . $v;
            }
         }
      }
}
return $url_normalized;
}
?>

crawl.php
<?php
function get_document($urltofetch){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urltofetch);

curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$str = curl_exec($ch);
curl_close($ch);
return $str;
}
function parsed_document_links($document_string){
$hrefs = array();
$doc = new DOMDocument();
@$doc->loadHTML($document_string);
$anchor_tags = $doc->getElementsByTagName('a');
foreach($anchor_tags as $anchor){
      $hrefs[] = $anchor->getAttribute('href');
}
return $hrefs;
}
function normalize_links($links_array){
global $exclude_terms_url;
$url_normalized = array();
foreach($links_array as $k=>$v){
      $skip_url = false;
      foreach($exclude_terms_url as $key=>$value){
         if(stristr($v,$value) !== false)
            $skip_url = true;
      }
      if(!$skip_url){
         $parsed_link = @parse_url($v);
         if(SINGLE_SITE){
            if(isset($parsed_link['scheme']) && $parsed_link['scheme'] == 'http'){
                  if(stristr($v,DOMAIN) !== false){
                     $url_normalized[] = $v;
                  }
            }else{
                  $url_normalized[] = URL_SEED . $v;
            }
         }
      }
}
return $url_normalized;
}
?>

[ 本帖最后由 workingbeijing 于 2008-3-4 09:58 編輯 ]