pengzhekuan
挖个坟...
这个网页使用了JavaScript进行渲染,所以直接使用rvest::read_html()
无法得到我们所看到的html。
此外,所需要爬取的文献标题和作者列表并不全部显示在页面中。而是通过不断点击show more而不断被加载。
所以,用常规的方法很难爬取页面。比较直接的解决办法是使用工具模拟浏览器的行为加载全部页面,并且模拟人工点击show more从而获取论文标题和作者。
因为不确定R有没有相同功能的包,这里给出Python的解决方法吧:
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
def get_html(url):
'''模拟浏览器加载页面并点击'''
browser = webdriver.Chrome()
browser.get(url)
time.sleep(5)
# 点击 Citations元素
browser.find_element_by_xpath("//div[contains(text(),'Citations')]").click()
print('点击Citations完成')
time.sleep(10)
# 点击show more
for elem in browser.find_elements_by_xpath("//span[contains(text(),'Show more')]"):
elem.click()
print('点击show more完成')
soup = BeautifulSoup(browser.page_source,'lxml')
return soup
def extract_page(soup):
'''解析html页面'''
dt_list = []
for j in soup.find_all(class_='nova-v-publication-item__body'):
paper_title = j.find(class_='nova-e-link nova-e-link--color-inherit nova-e-link--theme-bare')
if 'publication' in paper_title['href']:
title = paper_title.string
author_list = j.find_all(class_='nova-v-person-inline-item__fullname')
author_lst = [i.string for i in author_list]
author_lst = ','.join(author_lst)
dict_row = {'title': title,
'authors': author_lst}
dt_list.append(dict_row)
return dt_list
url = 'https://www.researchgate.net/publication/26787109_The_urban_heat_island_and_its_impact_on_heat_waves_and_human_health_in_Shanghai?tdsourcetag=s_pcqq_aiomsg'
soup_dic = get_html(url)
最后得到一份论文标题对应作者的dict:
[{'title': 'Influence of Urban Scale and Urban Expansion on the Urban Heat Island Effect in Metropolitan Areas: Case Study of Beijing-Tianjin-Hebei Urban Agglomeration',
'authors': 'Mingxing Chen,Zhou Yuan,Maogui Hu,Yaliu Zhou'},
{'title': 'Intraurban social risk and mortality patterns during extreme heat events: A case study of Moscow',
'authors': 'Stepan Zemtsov,Natalia Shartova,Mikhail Varentsov,Mikhail Grischchenko'},
{'title': 'Îlots de Chaleur Urbains et Ville: La Perception des Habitants',
'authors': 'Pia Monnier,Karine Dupre,Silvia Tavares'},
{'title': 'Tree crowns and their associated summertime microclimatic adjustment and thermal comfort improvement in urban parks in a subtropical city of China',
'authors': 'Jian Zhang,Zhonghua Gou'},
{'title': 'Multi-hazard climate risk projections for the United States',
'authors': 'Binita Kc,J. M. Shepherd,Anthony W. King,Cassandra Johnson Gaither'},
{'title': 'Surface Urban Heat Island in Middle City: Spatial and Temporal Characteristics',
'authors': 'Elis Alves,Max Anjos,Emerson Galvani'},
{'title': 'Socioeconomic level and associations between heat exposure and all-cause and cause-specific hospitalization in 1,814 Brazilian cities: A nationwide case-crossover study',
'authors': 'Rongbin Xu,Qi Zhao,Micheline S. Z. S. Coelho,Yuming Guo'},
{'title': "Exploring difference in land surface temperature between the city centres and urban expansion areas of China's major cities Exploring difference in land surface temperature between the city centres and urban expansion areas of China's major cities",
'authors': 'Jiyao Zhao,Le Yu,Yidi Xu,Peng'}
如果想转化成pd.DataFrame
的话:
paper_list = pd.DataFrame(soup_dic)
+----+---------------------------------------------------+---------------------------------------------------+
| | title | authors |
+----+---------------------------------------------------+---------------------------------------------------+
| 0 | Influence of Urban Scale and Urban Expansion o... | Mingxing Chen,Zhou Yuan,Maogui Hu,Yaliu Zhou |
+----+---------------------------------------------------+---------------------------------------------------+
| 1 | Intraurban social risk and mortality patterns ... | Stepan Zemtsov,Natalia Shartova,Mikhail Varent... |
+----+---------------------------------------------------+---------------------------------------------------+
| 2 | Îlots de Chaleur Urbains et Ville: La Percepti... | Pia Monnier,Karine Dupre,Silvia Tavares |
+----+---------------------------------------------------+---------------------------------------------------+
| 3 | Tree crowns and their associated summertime mi... | Jian Zhang,Zhonghua Gou |
+----+---------------------------------------------------+---------------------------------------------------+
| 4 | Multi-hazard climate risk projections for the ... | Binita Kc,J. M. Shepherd,Anthony W. King,Cassa... |
+----+---------------------------------------------------+---------------------------------------------------+
| 5 | Surface Urban Heat Island in Middle City: Spat... | Elis Alves,Max Anjos,Emerson Galvani |
+----+---------------------------------------------------+---------------------------------------------------+
| 6 | Socioeconomic level and associations between h... | Rongbin Xu,Qi Zhao,Micheline S. Z. S. Coelho,Y... |
+----+---------------------------------------------------+---------------------------------------------------+
| 7 | Exploring difference in land surface temperatu... | Jiyao Zhao,Le Yu,Yidi Xu,Peng |
+----+---------------------------------------------------+---------------------------------------------------+
| 8 | Thermal Infrared Images to Identify the Contri... | Floriberta Binarti,Pranowo Pranowo,Soesilo Boe... |
+----+---------------------------------------------------+---------------------------------------------------+
| 9 | Climate change and trend analysis of temperatu... | Zinabu Assefa Alemu,Michael Dioha |
+----+---------------------------------------------------+---------------------------------------------------+
| 10 | A study of the high temperature distribution a... | J.C. Ding,Z.K. Zhang,H. Xi,H.-M. Zhou |
+----+---------------------------------------------------+---------------------------------------------------+