libingfei
玩游戏玩的晚了些,看到你这个帖子,我突然想起一段痛苦的回忆,所以写些我的想法。
问题1,查了下这个post是存在的。
不过post后,是强行跳转到对应页面。所以post不直接返回数据。
例如:
http://xh.5156edu.com/index.php?f_key=%CD%F5&f_type=zi&SearchString.x=0&SearchString.y=0
是查找汉字[王],查询后返回302进入跳转页http://xh.5156edu.com/html3/13280.html
所以下面的语句是可以爬取对应网页的。
getURL("http://xh.5156edu.com/index.php?f_key=%CD%F5&f_type=zi&SearchString.x=0&SearchString.y=0",followlocation = TRUE,.encoding = 'UTF-8')
问题2,土豆ajax,也是需要post的,你查得不细而已,土豆评论的url是如下格式。
下面是无敌铁桥三的评论url;各字段都是啥含义,我没研究,你继续研究吧。
http://www.tudou.com/comments/itemnewcomment.srv?jsoncallback=module_data_comment_get&iid=132386962&page=1&rows=50&cmtid=0&tm=17&method=getHotCmt&ids=&charset=utf-8
爬下来的json是这种格式:
module_data_comment_get({"total":16,"tm":17,"popCmtArray":[],"data":[{"picUrl":"","location":"","status":0,"nickname":"小土豆374538459","mark":0,"itemId":132386962,"userpic":"http:\/\/u2.tdimg.com\/7\/86\/64\/11083029128378728231430700528796645897.jpg","verifyType":0,"headlink":"","commentId":54451012,"content":"拜师的那小兄弟该不会是黄飞鸿吧。铁桥三不能只收他一个做徒弟,又不好推辞,应该就是林福成收下做徒弟了。相传铁桥三把铁线拳传给林福成,林福成再传给了黄飞鸿。","userID":374538459,"username":"_374538459","publish_time":"1427303579000","motherId":0,"isvip":false,"ucode":"fLJYAjEyb6I"},{"picUrl":"","location":"","status":0,"nickname":"小土豆485622308","mark":0,"itemId":132386962,"userpic":"http:\/\/u1.tdimg.com\/2\/216\/140\/93920982148704719938873924700860276152.jpg","verifyType":0,"headlink":"","commentId":54451220,"content":"江林儿见到梁坤不马上认,我想江林儿离死差不多了。应该是要死时梁坤整好看到然后告诉他瓷谱和口绝的地方。是这样吗?导演","userID":485622308,"username":"_485622308","publish_time":"1427308356000","motherId":0,"isvip":false,"ucode":"ldS-OmMj3Wc"},{"picUrl":"","location":"","status":0,"nickname":"小土豆390314410","mark":0,"itemId":132386962,"userpic":"http:\/\/u2.tdimg.com\/5\/132\/91\/_156866617227494743304187072709420196895.jpg","verifyType":0,"headlink":"","commentId":54451145,"content":"和不了,太不小心了","userID":390314410,"username":"_390314410","publish_time":"1427306582000","motherId":0,"isvip":false,"ucode":"5VEfoPyab8w"},{"picUrl":"","location":"","status":0,"nickname":"不可救药FE33D","mark":0,"itemId":132386962,"userpic":"http:\/\/u4.tdimg.com\/u\/U-07.gif","verifyType":0,"headlink":"","commentId":54451085,"content":"傻啊,是鸿鸿他爹啊","userID":370940542,"motCmt":{"picUrl":"","location":"","status":0,"nickname":"小土豆374538459","mark":0,"itemId":132386962,"userpic":"http:\/\/u2.tdimg.com\/7\/86\/64\/11083029128378728231430700528796645897.jpg","verifyType":0,"headlink":"","commentId":54451012,"content":"拜师的那小兄弟该不会是黄飞鸿吧。铁桥三不能只收他一个做徒弟,又不好推辞,应该就是林福成收下做徒弟了。相传铁桥三把铁线拳传给林福成,林福成再传给了黄飞鸿。","userID":374538459,"username":"_374538459","publish_time":"1427303579000","motherId":0,"isvip":false,"ucode":"fLJYAjEyb6I"},"username":"_370940542","publish_time":"1427304978000","motherId":54451012,"isvip":false,"ucode":"LSPScL9b4Y0"},{"picUrl":"","location":"","status":0,"nickname":"小土豆724670386","mark":0,"itemId":132386962,"userpic":"http:\/\/u3.tdimg.com\/2\/195\/211\/_56172926892846797576761395927819941282.jpg","verifyType":0,"headlink":"","commentId":54451070,"content":"是的话也是黄麒英啊","userID":724670386,"motCmt":{"picUrl":"","location":"","status":0,"nickname":"小土豆374538459","mark":0,"itemId":132386962,"userpic":"http:\/\/u2.tdimg.com\/7\/86\/64\/11083029128378728231430700528796645897.jpg","verifyType":0,"headlink":"","commentId":54451012,"content":"拜师的那小兄弟该不会是黄飞鸿吧。铁桥三不能只收他一个做徒弟,又不好推辞,应该就是林福成收下做徒弟了。相传铁桥三把铁线拳传给林福成,林福成再传给了黄飞鸿。","userID":374538459,"username":"_374538459","publish_time":"1427303579000","motherId":0,"isvip":false,"ucod.......................
问题三:百度的数字是图片形式,爬取百度指数,一个笨方法,你需要些样本图片最比对,将爬下来的的图片映射到数字上。怎么想都很麻烦。也许还有更好的方法,我还没想到。