使用beautfulSoup抓取網(wǎng)頁數(shù)據(jù)并寫入txt文件失敗

maple412

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import urllib2
import re

link=''
path=r'E:/pydownload/1.txt'
f=open(path,'wb+')
for i in range(1,150):
    print "it is download the %d page:" % i
    new=link + str(i) + '.'+'shtml'
    req=urllib2.Request(new)
    fd=urllib2.urlopen(req)
    soup=BeautifulSoup(fd.read(),from_encoding="utf-8"
    ret=soup.find(id="content_1"
    for r in ret.stripped_strings:
        f.write(r)

f.close()
網(wǎng)頁的內(nèi)容是中文的。提示如下錯誤：
Traceback (most recent call last):
  File "E:\py_prj\test1.py", line 17, in <module>
    f.write(r)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(12

ma__

1. reload(sys)
   
2. sys.setdefaultencoding( "utf-8" )

復制代碼

maple412

回復 2# ma__
沒用，一樣的效果

ma__

1. # -*- coding: utf-8 -*-
   
2. from bs4 import BeautifulSoup
   
3. import urllib2
   
4. import re,sys
   
5. reload(sys)
   
6. sys.setdefaultencoding( "utf-8" )
   
7. 
   
8. link='http:     //movie.douban.com/'
   
9. path=r'1.txt'
   
10. f=open(path,'wb+')
    
11. # print "it is download the %d page:" % i
    
12. # new=link + str(i) + '.'+'shtml'
    
13. req=urllib2.Request(link)
    
14. fd=urllib2.urlopen(req)
    
15. soup=BeautifulSoup(fd.read(),from_encoding="utf-8")
    
16. ret=soup.find(id="top-nav-appintro")
    
17. for r in ret.stripped_strings:
    
18.         f.write(r)
    
19.         print r
    
20. # f.close()
    

復制代碼

To get rid of this warning, change this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "lxml"

豆瓣
3.0
和有趣的人做有趣的事
掃碼直接下載
iPhone
·
Android
為什么我可以

ma__

r是unicode
那你寫之前加行
r=r.encode('utf-8','ignore')
吧