I'm trying to call a number of spiders from a shell script like so:
#!/bin/bash
source /home/pi/.bashrc
source /usr/local/bin/virtualenvwrapper.sh
workon email-scraper
cdvirtualenv
#Change to dir containing scrapy.cfg
cd ./sitecode/jobscraper
pwd
scrapy crawl site1
scrapy crawl site2
scrapy crawl site3
# Email me the jobs.csv file
python '/home/pi/Documents/jobscraper/scripts/jobs-mail.py'
# Delete the file so a new one is created on next scrape
# sudo rm /home/pi/Documents/jobscraper/csv/jobs.csv
The only part it executes properly is the python script at the end, which emails me an empty csv file as none of the scrapers have run. I get the following output when running the above script from BASH
(email-scraper)pi@raspberrypi ~/.virtualenvs/email-scraper/sitecode $ sudo sh runspiders.sh
ERROR: Environment 'email-scraper' does not exist. Create it with 'mkvirtualenv email-scraper'.
ERROR: no virtualenv active, or active virtualenv is missing
runspiders.sh: line 9: cd: ./sitecode/jobscraper: No such file or directory
/home/pi/.virtualenvs/email-scraper/sitecode
runspiders.sh: line 13: scrapy: command not found
runspiders.sh: line 14: scrapy: command not found
runspiders.sh: line 15: scrapy: command not found
runspiders.sh: line 16: scrapy: command not found
runspiders.sh: line 17: scrapy: command not found
runspiders.sh: line 18: scrapy: command not found
runspiders.sh: line 19: scrapy: command not found
runspiders.sh: line 20: scrapy: command not found
runspiders.sh: line 21: scrapy: command not found
runspiders.sh: line 22: scrapy: command not found
I'm very new to shell scripting. Can anyone illustrate how to make sure I activate the virtualenv and change into the correct dir before calling the first spider?
Edit In reply to @konsolebox
This is how I would run the scraper manually from the home dir:
First, source .bashrc as the raspberry pi doesn't do this automatically for some reason.
source .bashrc
This allows me access to virtualenvwrapper. I can then
pi@raspberrypi ~ $ workon email-scraper
(email-scraper)pi@raspberrypi ~ $ cdvirtualenv
which puts me in the virtualenv project directory /home/pi/.virtualenvs/email-scraper
I then do
cd sitecode/jobscraper
and ls -al puts me in the dir with access to scrapy.cfg, which I need in order to run the scraper.
drwxr-xr-x 3 pi pi 4096 Sep 9 19:40 .
drwxr-xr-x 5 pi pi 4096 Sep 8 19:41 ..
drwxr-xr-x 3 pi pi 4096 Sep 8 14:59 jobscraper
-rwxr-xr-x 1 pi pi 632 Sep 8 22:18 runspiders.sh
-rw-r--r-- 1 root root 12288 Sep 9 19:40 .runspiders.sh.swp
-rw-r--r-- 1 pi pi 381 Sep 7 23:34 scrapy.cfg
I can then do scrapy crawl site1 to run the scraper.