crawler/README at master · chw120/crawler · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
VERSION 1.0

Prerequisites:
    OS:Linux
    Python2.7, pip
       for windows:
          +install: pywin32, visual c++ 9.0

    render Javascript:
      splash (a Javascript rendering service): Docker
      scrapyjs
      start splash service
        sudo docker run -p 8050:8050 scrapinghub/splash

install scrapy:
    pip install scrapy

Start a scrapy project
    use cmd moving the specified folder;
    scrapy startproject projectname.
create spidername.py under spiders/

add crawling logic in a scrapy spider
    items.py: define data schema.
    pipelines.py: process data after fetching.
    settings.py: define configurations, eg: use proxy.
    spider: logic.(normally, each website respectively has one spider.)

run spider
    cd projectname
    scrapy crawl name


Tips:
When setting SPLASH_URL in settings.py, run $docker-machine ip to get the ip address if using Docker.