Go爬虫colly官方示例二十一【shopify_sitemap】- 使用XPath请求抓取shopify网站地图里url

转载请标明出处：
http://blog.csdn.net/qq_27818541/article/details/112587216
本文出自:【BigManing的博客】

示例介绍

1. 访问shopify网站地图url https://www.shopify.com/sitemap.xml
2. c.OnXML 爬取 //urlset/url/loc 对应的值
3. 组装到集合中
4. 打印

示例代码

package main

import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	// Array containing all the known URLs in a sitemap
	knownUrls := []string{}

	// Create a Collector specifically for Shopify
	//设置domain
	c := colly.NewCollector(colly.AllowedDomains("www.shopify.com"))

	// Create a callback on the XPath query searching for the URLs    
	//由于数据结构是xml，所以使用XPath查询 爬取每个url
	c.OnXML("//urlset/url/loc", func(e *colly.XMLElement) {
		knownUrls = append(knownUrls, e.Text)
	})

	// Start the collector 
	//访问url
	c.Visit("https://www.shopify.com/sitemap.xml")

	fmt.Println("All known URLs:")
	for _, url := range knownUrls {
		fmt.Println("\t", url)
	} 
	// 打印数据
	fmt.Println("Collected", len(knownUrls), "URLs")
}

名词解释

Sitemap 简介

可方便网站管理员通知搜索引擎他们网站上有哪些可供抓取的网页。最简单的 Sitemap 形式，就是XML 文件，在其中列出网站中的网址以及关于每个网址的其他元数据（上次更新的时间、更改的频率以及相对于网站上其他网址的重要程度为何等），以便搜索引擎可以更加智能地抓取网站。

XPath

XPath，全称 XML Path Language，即 XML 路径语言，它是一门在 XML 文档中查找信息的语言。最初是用来搜寻 XML 文档的，但同样适用于 HTML 文档的搜索。

输出

All known URLs:
         https://www.shopify.com/
         https://www.shopify.com/3d
         https://www.shopify.com/about
         https://www.shopify.com/about/economic-impact
         https://www.shopify.com/about/economic-impact/canada
         https://www.shopify.com/about/economic-impact/global
         https://www.shopify.com/about/economic-impact/individual
         https://www.shopify.com/about/economic-impact/regional
         https://www.shopify.com/about/environment
         https://www.shopify.com/about/environment/sustainability-fund
         https://www.shopify.com/about/environment/sustainability-fund/biomass
         https://www.shopify.com/about/environment/sustainability-fund/collaboration
   
         https://www.shopify.com/success-stories/novo-watch
         https://www.shopify.com/success-stories/modern-market
         https://www.shopify.com/success-stories/datsusara
         https://www.shopify.com/success-stories/catherine-zoraida
         https://www.shopify.com/success-stories/orange-and-park
         https://www.shopify.com/success-stories/nj-swingsets
         https://www.shopify.com/success-stories/bija-bhar
         https://www.shopify.com/success-stories/groomstand
         https://www.shopify.com/success-stories/regal-gentlemen
         https://www.shopify.com/success-stories/lightbox-photography-cards
         https://www.shopify.com/success-stories/suffolk-latch-company
         https://www.shopify.com/success-stories/one-tribe-apparel
         https://www.shopify.com/success-stories/tikker-watches
         https://www.shopify.com/success-stories/mstrmnd-collective
         https://www.shopify.com/success-stories/old-world-kitchen-by-polder-s-old-world-market
         https://www.shopify.com/success-stories/growing-footprints
         https://www.shopify.com/success-stories/love-that-bag
         https://www.shopify.com/success-stories/sweet-spot-skirts
         https://www.shopify.com/success-stories/christmas-shirt-company
         https://www.shopify.com/success-stories/unsung-hero-apparel
         https://www.shopify.com/success-stories/enjoy-meditation
         https://www.shopify.com/success-stories/fragrance-buy
         https://www.shopify.com/success-stories/laguna-salt-company
         https://www.shopify.com/success-stories/sube
         https://www.shopify.com/success-stories/tiny-devotions
         https://www.shopify.com/success-stories/modish-store
         https://www.shopify.com/success-stories/bogart
         https://www.shopify.com/success-stories/the-future-kept
         https://www.shopify.com/success-stories/longsands-clothing-co
         https://www.shopify.com/success-stories/maitri-learning
         https://www.shopify.com/success-stories/koogal
         https://www.shopify.com/success-stories/mezel-mods
         https://www.shopify.com/success-stories/shoreline-music
         https://www.shopify.com/success-stories/the-last-hunt
         https://www.shopify.com/success-stories/krama-wheel
         https://www.shopify.com/success-stories/girly-go-garter
         https://www.shopify.com/success-stories/dewerstone
         https://www.shopify.com/success-stories/lotus-belle-usa
         https://www.shopify.com/success-stories/all-mountain-style
         https://www.shopify.com/success-stories/mountain-standard
         https://www.shopify.com/success-stories/jones-of-boerum-hill
         https://www.shopify.com/success-stories/kristin-schmucker-shop
         https://www.shopify.com/success-stories/brooklyn-bambini
         https://www.shopify.com/success-stories/the-comfortable-boxers-co
         https://www.shopify.com/success-stories/latched-mama
         https://www.shopify.com/success-stories/floor-tape-store
         https://www.shopify.com/success-stories/mukhayoga
         https://www.shopify.com/success-stories/tilit-chef-goods
         https://www.shopify.com/success-stories/gem-chocolates
         https://www.shopify.com/success-stories/plant-with-purpose
         https://www.shopify.com/success-stories/brownfields-shop
         https://www.shopify.com/success-stories/notabag-tote
         https://www.shopify.com/success-stories/project-repat
         https://www.shopify.com/success-stories/cape-whoopies
         https://www.shopify.com/success-stories/be-green-packaging-store
         https://www.shopify.com/success-stories/think-labs
         https://www.shopify.com/success-stories/vlu-accessories
         https://www.shopify.com/success-stories/skin-owl-oil
         https://www.shopify.com/success-stories/desires-by-mikolay
         https://www.shopify.com/success-stories/vertty
         https://www.shopify.com/success-stories/jerky-spot
         https://www.shopify.com/success-stories/the-yes-bar
         https://www.shopify.com/success-stories/holy-crap
         https://www.shopify.com/success-stories/mud-gear
         https://www.shopify.com/success-stories/econica-lingerie
         https://www.shopify.com/success-stories/not-another-bunch-of-flowers
         https://www.shopify.com/success-stories/the-uncommon-green
         https://www.shopify.com/success-stories/qubits-toy
         https://www.shopify.com/success-stories/shinesty-clothing
         https://www.shopify.com/success-stories/betsy-and-iya
         https://www.shopify.com/success-stories/quest-award-shop
         https://www.shopify.com/success-stories/retro-city-sunglasses
         https://www.shopify.com/success-stories/lets-hug-it-out
         https://www.shopify.com/success-stories/leaves-of-trees
         https://www.shopify.com/success-stories/biko-jewelry
         https://www.shopify.com/success-stories/timber-plates
         https://www.shopify.com/success-stories/baguera
         https://www.shopify.com/success-stories/chappy-happy
         https://www.shopify.com/success-stories/digital-tintypes
         https://www.shopify.com/success-stories/the-net-return
         https://www.shopify.com/success-stories/me-cam
         https://www.shopify.com/success-stories/tigerlily-sandals
         https://www.shopify.com/success-stories/warren-london-dog-products
         https://www.shopify.com/success-stories/famous-games-co
         https://www.shopify.com/success-stories/outline-the-sky
         https://www.shopify.com/success-stories/buiced
         https://www.shopify.com/success-stories/encircled
         https://www.shopify.com/success-stories/soaring-heart
         https://www.shopify.com/success-stories/desert-farms-camel-milk
         https://www.shopify.com/success-stories/chemo-cozy
         https://www.shopify.com/success-stories/humble-hilo
         https://www.shopify.com/success-stories/oxygen-plus
         https://www.shopify.com/success-stories/beauty-empire
         https://www.shopify.com/success-stories/sockhopny
         https://www.shopify.com/success-stories/matter-goods
         https://www.shopify.com/success-stories/encarded-custom-cards
         https://www.shopify.com/success-stories/adorable-essentials
         https://www.shopify.com/success-stories/equss
         https://www.shopify.com/success-stories/lazy-fruits
         https://www.shopify.com/success-stories/mixed-made
         https://www.shopify.com/success-stories/wish-box
         https://www.shopify.com/success-stories/esqido-lashes
         https://www.shopify.com/success-stories/brooklyn-delhi
         https://www.shopify.com/success-stories/same-sky
         https://www.shopify.com/success-stories/stayhold-stand
         https://www.shopify.com/success-stories/top-anchor-quilting
         https://www.shopify.com/success-stories/kigurumi
         https://www.shopify.com/success-stories/fantasy-jocks
         https://www.shopify.com/success-stories/my-swim-stuff
         https://www.shopify.com/success-stories/united-by-blue
         https://www.shopify.com/success-stories/shop-lovely
         https://www.shopify.com/success-stories/lilitab
         https://www.shopify.com/success-stories/beautiful-photo-props
         https://www.shopify.com/success-stories/top-and-derby
         https://www.shopify.com/success-stories/3d-universe
         https://www.shopify.com/success-stories/proof-eyewear
         https://www.shopify.com/success-stories/kitchen-safe
         https://www.shopify.com/success-stories/suitable
         https://www.shopify.com/success-stories/64-facets
         https://www.shopify.com/success-stories/panda-sunglasses
         https://www.shopify.com/success-stories/slide-belts
         https://www.shopify.com/success-stories/strobepro-studio-lighting
         https://www.shopify.com/success-stories/caljava-online
         https://www.shopify.com/success-stories/hammock-universe
         https://www.shopify.com/success-stories/get-back-supply-co
         https://www.shopify.com/success-stories/packed-party
         https://www.shopify.com/success-stories/tradlands
       
       ....
       
Collected 1251 URLs