火车采集器软件交流官方论坛

 找回密码
 加入会员
搜索
火车采集器V9版免费下载火车浏览器 - 可视采集,万能群发,全自动脚本工具
查看: 4480|回复: 9

地址有规则,无法抓去源码采集

[复制链接]
发表于 2013-9-18 23:19:28 | 显示全部楼层 |阅读模式
本帖最后由 commonyang 于 2013-9-18 23:20 编辑

地址
如上地址,因无法看到网页源码,而无法采集。除了通过源码,不知道有没其他方法可以采集的?
  1. http://www.google.com/cse?cx=partner-pub-8642343895325952%3A9186677094&cof=FORID%3A10&ie=UTF-8&q=cosmetics&sa=Search&ad=n9&num=10&rurl=http%3A%2F%2Fwww.articledashboard.com%2Fgsearch.html%3Fcx%3Dpartner-pub-8642343895325952%253A9186677094%26cof%3DFORID%253A10%26ie%3DUTF-8%26q%3Dcosmetics%26sa%3DSearch&siteurl=http%3A%2F%2Fwww.articledashboard.com%2F#gsc.tab=0&gsc.q=cosmetics&gsc.page=1
复制代码
发表于 2013-9-19 16:33:32 | 显示全部楼层
抓包, 或者切换为手机UA后再请求看源码, 源码肯定看得到的, 他应该是封到JS里,你抓包工具里找找.
发表于 2013-9-23 09:40:33 | 显示全部楼层
发表于 2013-9-23 09:41:03 | 显示全部楼层
直接贴上源码
  1. // API callback
  2. google.search.Search.apiary14457({"cursor":{"currentPageIndex":0,"estimatedResultCount":"3360","moreResultsUrl":"http://www.google.com/cse?oe=utf8&ie=utf8&source=uds&q=cosmetics&safe=active&start=0&cx=partner-pub-8642343895325952:9186677094","resultCount":"3,360","searchResultTime":"0.11","pages":[{"label":1,"start":"0"},{"label":2,"start":"10"},{"label":3,"start":"20"},{"label":4,"start":"30"},{"label":5,"start":"40"},{"label":6,"start":"50"},{"label":7,"start":"60"},{"label":8,"start":"70"},{"label":9,"start":"80"},{"label":10,"start":"90"}]},"context":{"title":"2011 Search","facets":[]},"results":[{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:C5BLZfbPk94J:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Mineral-Cosmetics-VS-Traditional-Cosmetics/2520037&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CAcQFjAA&client=internal-uds-cse&usg=AFQjCNH-q9fX4tcjlHM2Gqm5DPLefXVsyA","content":"Mineral \u003cb\u003eCosmetics\u003c/b\u003e VS Traditional \u003cb\u003eCosmetics\u003c/b\u003e Mineral make-up has been around   for more than thirty years already. However, it is only recently that mineral \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Mineral Cosmetics VS Traditional Cosmetics Mineral make-up has been around   for more than thirty years already. However, it is only recently that mineral ...","formattedUrl":"www.articledashboard.com/Article/...\u003cb\u003eCosmetics\u003c/b\u003e...\u003cb\u003eCosmetics\u003c/b\u003e/2520037","title":"Mineral \u003cb\u003eCosmetics\u003c/b\u003e Vs Traditional \u003cb\u003eCosmetics\u003c/b\u003e - Article Dashboard","titleNoFormatting":"Mineral Cosmetics Vs Traditional Cosmetics - Article Dashboard","unescapedUrl":"http://www.articledashboard.com/Article/Mineral-Cosmetics-VS-Traditional-Cosmetics/2520037","url":"http://www.articledashboard.com/Article/Mineral-Cosmetics-VS-Traditional-Cosmetics/2520037","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:fvUGn-IsEqwJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/How-Safe-are-Your-Cosmetics-Check-for-These-Three-Unwanted-Ingredients/2933769&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CAkQFjAB&client=internal-uds-cse&usg=AFQjCNGMroMtYD9yMVEuBj7h4S964Ief3Q","content":"Putting on makeup makes many women feel luxurious. Time spent pampering   oneself in front of a mirror can help boost confidence. Yet, considering that many   \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Putting on makeup makes many women feel luxurious. Time spent pampering   oneself in front of a mirror can help boost confidence. Yet, considering that many   ...","formattedUrl":"www.articledashboard.com/Article/How...\u003cb\u003eCosmetics\u003c/b\u003e.../2933769","title":"How Safe Are Your \u003cb\u003eCosmetics\u003c/b\u003e?: Check For These Three Unwanted \u003cb\u003e...\u003c/b\u003e","titleNoFormatting":"How Safe Are Your Cosmetics?: Check For These Three Unwanted ...","unescapedUrl":"http://www.articledashboard.com/Article/How-Safe-are-Your-Cosmetics-Check-for-These-Three-Unwanted-Ingredients/2933769","url":"http://www.articledashboard.com/Article/How-Safe-are-Your-Cosmetics-Check-for-These-Three-Unwanted-Ingredients/2933769","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:Fh26sZvIBZ4J:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Bringing-Cosmetics-When-Traveling-10-Things-to-Remember/2923844&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CAsQFjAC&client=internal-uds-cse&usg=AFQjCNEoR8bhBF6QO9vq1daViZnUvKFeAA","content":"Women cannot leave home without bringing their make-up kits and some   personal care stuff. Whether they are heading for work or going for a vacation,   they \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Women cannot leave home without bringing their make-up kits and some   personal care stuff. Whether they are heading for work or going for a vacation,   they ...","formattedUrl":"www.articledashboard.com/Article/Bringing-\u003cb\u003eCosmetics\u003c/b\u003e.../2923844","title":"Bringing \u003cb\u003eCosmetics\u003c/b\u003e When Traveling: 10 Things To Remember","titleNoFormatting":"Bringing Cosmetics When Traveling: 10 Things To Remember","unescapedUrl":"http://www.articledashboard.com/Article/Bringing-Cosmetics-When-Traveling-10-Things-to-Remember/2923844","url":"http://www.articledashboard.com/Article/Bringing-Cosmetics-When-Traveling-10-Things-to-Remember/2923844","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:IizihkbwJZwJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Create-sublime-HD-photos-of-cosmetics-products-with-PackshotCreator/2391576&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CA0QFjAD&client=internal-uds-cse&usg=AFQjCNHbIVY6fSL7tMsVvVpc2kmhCwm11Q","content":"\u003cb\u003eCosmetics\u003c/b\u003e photos are primordial in this world of fashion and beauty. In every   newspaper or in every magazine which you open and read, there is bound to be   at \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Cosmetics photos are primordial in this world of fashion and beauty. In every   newspaper or in every magazine which you open and read, there is bound to be   at ...","formattedUrl":"www.articledashboard.com/Article/Create...\u003cb\u003ecosmetics\u003c/b\u003e.../2391576","title":"Create Sublime Hd Photos Of \u003cb\u003eCosmetics\u003c/b\u003e Products With \u003cb\u003e...\u003c/b\u003e","titleNoFormatting":"Create Sublime Hd Photos Of Cosmetics Products With ...","unescapedUrl":"http://www.articledashboard.com/Article/Create-sublime-HD-photos-of-cosmetics-products-with-PackshotCreator/2391576","url":"http://www.articledashboard.com/Article/Create-sublime-HD-photos-of-cosmetics-products-with-PackshotCreator/2391576","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:ef56QRoKZisJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Selling-Cosmetics-And-How-To-Be-Successful/2204928&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CA8QFjAE&client=internal-uds-cse&usg=AFQjCNFPHkTK8jnEtDxgOns_Q4Qk3eLo_g","content":"One of the most universal traits of us women is in caring for our beauty, not all,   but in the majority of cases, we will do whatever it takes just to be beautiful in so \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"One of the most universal traits of us women is in caring for our beauty, not all,   but in the majority of cases, we will do whatever it takes just to be beautiful in so ...","formattedUrl":"www.articledashboard.com/Article/Selling-\u003cb\u003eCosmetics\u003c/b\u003e.../2204928","title":"Selling \u003cb\u003eCosmetics\u003c/b\u003e And How To Be Successful","titleNoFormatting":"Selling Cosmetics And How To Be Successful","unescapedUrl":"http://www.articledashboard.com/Article/Selling-Cosmetics-And-How-To-Be-Successful/2204928","url":"http://www.articledashboard.com/Article/Selling-Cosmetics-And-How-To-Be-Successful/2204928","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:sYeJAcEFdu0J:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Botox-Has-Revolutionized-the-Cosmetics-Industry/2037510&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CBEQFjAF&client=internal-uds-cse&usg=AFQjCNHrXvLVOYhsaf0R8Frwiq_w_6fnHA","content":"Since the late 1980's, Botox has made it so that millions of people to smile   without annoying wrinkles, crow's feet, and frown lines. Originally considered \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Since the late 1980's, Botox has made it so that millions of people to smile   without annoying wrinkles, crow's feet, and frown lines. Originally considered ...","formattedUrl":"www.articledashboard.com/Article/Botox-Has...\u003cb\u003eCosmetics\u003c/b\u003e.../2037510","title":"Botox Has Revolutionized The \u003cb\u003eCosmetics\u003c/b\u003e Industry - Article Dashboard","titleNoFormatting":"Botox Has Revolutionized The Cosmetics Industry - Article Dashboard","unescapedUrl":"http://www.articledashboard.com/Article/Botox-Has-Revolutionized-the-Cosmetics-Industry/2037510","url":"http://www.articledashboard.com/Article/Botox-Has-Revolutionized-the-Cosmetics-Industry/2037510","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:q7n1Evn2GMEJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Water-Mineral-or-Oil-Comparing-the-Base-Ingredients-in-Cosmetics/703944&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CBMQFjAG&client=internal-uds-cse&usg=AFQjCNHKep_W1oAPa9iRMGsld_iuJOYbgQ","content":"There are a number of ingredients which are used in \u003cb\u003ecosmetics\u003c/b\u003e, with various   types of ingredients being used to create vastly different colors and textures once   \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"There are a number of ingredients which are used in cosmetics, with various   types of ingredients being used to create vastly different colors and textures once   ...","formattedUrl":"www.articledashboard.com/Article/Water...\u003cb\u003eCosmetics\u003c/b\u003e/703944","title":"Water, Mineral Or Oil? Comparing The Base Ingredients In \u003cb\u003eCosmetics\u003c/b\u003e","titleNoFormatting":"Water, Mineral Or Oil? Comparing The Base Ingredients In Cosmetics","unescapedUrl":"http://www.articledashboard.com/Article/Water-Mineral-or-Oil-Comparing-the-Base-Ingredients-in-Cosmetics/703944","url":"http://www.articledashboard.com/Article/Water-Mineral-or-Oil-Comparing-the-Base-Ingredients-in-Cosmetics/703944","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:J-GglgujQ70J:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/New-Exclusive-Cosmetics-Wholesale-Products/1932374&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CBUQFjAH&client=internal-uds-cse&usg=AFQjCNFkoY79emaydqvPtHCOG2QnzfDGxw","content":"Women all over the world work hard to look great. This is particularly true as a   woman ages and she has to turn to \u003cb\u003ecosmetics\u003c/b\u003e products to help her not only look \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Women all over the world work hard to look great. This is particularly true as a   woman ages and she has to turn to cosmetics products to help her not only look ...","formattedUrl":"www.articledashboard.com/Article/New...\u003cb\u003eCosmetics\u003c/b\u003e.../1932374","title":"New Exclusive \u003cb\u003eCosmetics\u003c/b\u003e Wholesale Products - Article Dashboard","titleNoFormatting":"New Exclusive Cosmetics Wholesale Products - Article Dashboard","unescapedUrl":"http://www.articledashboard.com/Article/New-Exclusive-Cosmetics-Wholesale-Products/1932374","url":"http://www.articledashboard.com/Article/New-Exclusive-Cosmetics-Wholesale-Products/1932374","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:fCvcXmil97AJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Mary-Kay-Cosmetics-Enriching-Your-Way-To-Success/2945612&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CBcQFjAI&client=internal-uds-cse&usg=AFQjCNEu1XileMErMP390okK6VEd8GfTqA","content":"Mary Kay \u003cb\u003eCosmetics\u003c/b\u003e is a company that was started in 1963, offering \u003cb\u003ecosmetics\u003c/b\u003e   and similar products to men and women. It has dedicated much time for research   \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"Mary Kay Cosmetics is a company that was started in 1963, offering cosmetics   and similar products to men and women. It has dedicated much time for research   ...","formattedUrl":"www.articledashboard.com/Article/Mary...\u003cb\u003eCosmetics\u003c/b\u003e.../2945612","title":"Mary Kay \u003cb\u003eCosmetics\u003c/b\u003e – Enriching Your Way To Success","titleNoFormatting":"Mary Kay Cosmetics – Enriching Your Way To Success","unescapedUrl":"http://www.articledashboard.com/Article/Mary-Kay-Cosmetics-Enriching-Your-Way-To-Success/2945612","url":"http://www.articledashboard.com/Article/Mary-Kay-Cosmetics-Enriching-Your-Way-To-Success/2945612","visibleUrl":"www.articledashboard.com"},{"GsearchResultClass":"GwebSearch","cacheUrl":"http://www.google.com/search?q=cache:mNpC_AepTeIJ:www.articledashboard.com","clicktrackUrl":"https://www.google.com/url?q=http://www.articledashboard.com/Article/Do-Cosmetics-Causes-Acne/46838&sa=U&ei=Epw_UrDtIoSPrgeoyYGYBQ&ved=0CBkQFjAJ&client=internal-uds-cse&usg=AFQjCNFgAk_gtLSHY-7mck1PGLqiAfPOAg","content":"For decades, dermatologists and cosmetologists alike have debated the effects of   \u003cb\u003ecosmetics\u003c/b\u003e on the skin, particularly in acne sufferers. Make-up has often been \u003cb\u003e...\u003c/b\u003e","contentNoFormatting":"For decades, dermatologists and cosmetologists alike have debated the effects of   cosmetics on the skin, particularly in acne sufferers. Make-up has often been ...","formattedUrl":"www.articledashboard.com/Article/Do-\u003cb\u003eCosmetics\u003c/b\u003e-Causes.../46838","title":"Do \u003cb\u003eCosmetics\u003c/b\u003e Causes Acne?","titleNoFormatting":"Do Cosmetics Causes Acne?","unescapedUrl":"http://www.articledashboard.com/Article/Do-Cosmetics-Causes-Acne/46838","url":"http://www.articledashboard.com/Article/Do-Cosmetics-Causes-Acne/46838","visibleUrl":"www.articledashboard.com"}]});
复制代码
发表于 2013-9-23 10:14:03 | 显示全部楼层
lmj243 发表于 2013-9-23 09:41
直接贴上源码

http://bbs.locoy.com/spider-135232-1-1.html

针对这个做了个教程
 楼主| 发表于 2013-9-29 01:21:56 | 显示全部楼层
lmj243 发表于 2013-9-23 10:14
http://bbs.locoy.com/spider-135232-1-1.html

针对这个做了个教程

谢谢哈,一直没空过来,今天看到了你的留言,表示感谢和敬意。
 楼主| 发表于 2013-9-29 17:25:42 | 显示全部楼层
lmj243 发表于 2013-9-23 09:40
https://www.googleapis.com/customsearch/v1element?key=AIzaSyCVAXiUzRYsML1Pv6RwSG1gunmMikTzQqY&rsz=fi ...

可否指点一下,这个链接是在哪里找到的。我不知道为什么,老是找不到。
 楼主| 发表于 2013-9-29 17:27:52 | 显示全部楼层
 楼主| 发表于 2013-9-29 18:47:15 | 显示全部楼层
commonyang 发表于 2013-9-29 17:25
可否指点一下,这个链接是在哪里找到的。我不知道为什么,老是找不到。

刚刚找到了,原来是用firefox浏览器,fiddler抓不到firfox的内容。之后用ie再开VPN就可以正常查看到内容了。googleapis.com就可以正常反馈数据了。
发表于 2014-9-24 18:08:52 | 显示全部楼层
我看看 话说真能采集?
您需要登录后才可以回帖 登录 | 加入会员

本版积分规则

QQ|手机版|Archiver|火车采集器官方站 ( 皖ICP备06000549 )

GMT+8, 2024-11-25 07:40

Powered by Discuz! X3.4

Copyright © 2001-2020, Tencent Cloud.

快速回复 返回顶部 返回列表