Setup Developement Environment

I decide to spin up 1 Digital Ocean VPS to host my projects for now. This can be easily done with a Terraform file I used before:

variable "do_token" {
  default="####################################"
}

provider "digitalocean" {
  token = "${var.do_token}"
}

resource "digitalocean_droplet" "a1" {
  image  = "ubuntu-16-04-x64"
  name   = "a1"
  region = "tor1"
  size   = "1gb"
  #size   = "4gb"
#   user_data = <<EOF
# #!/bin/bash
# apt-get -y update
# apt-get -y install python
# EOF
  ssh_keys = ["###############################"]
}

resource "digitalocean_floating_ip" "myip" {
  region = "tor1"
  droplet_id = "${digitalocean_droplet.a1.id}"
}

output "ip" {
  value = "${digitalocean_floating_ip.myip.ip_address}"
}

The command to run is terraform apply.

Done!

---

The ip is 174.138.114.17. I'll add an entry to my host file for now. I will probably add a sub-domain for it soon so it's accessible by domain name:

174.138.114.17 marathon

---

We will use Ansible to do configuration management. First, Python is required by Ansible but is not installed in the default DO Ubuntu image. Let's install it using this playbook:

- hosts: all
  gather_facts: False
  
  tasks:
  - name: install python 2
    raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

The command to run is ansible-playbook ansible-init.yml -u root -i marathon,.

Success!

We will also install Docker suite on the server, so we can push our docker deployment there. A search on Ansible Galaxy shows that somebody has already created a darn good docker role with thousands of downloads. Let's use that:

- hosts: all
  roles:
    - mongrelion.docker

Then run ansible-galaxy install mongrelion.docker and then ansible-playbook -vvv init-docker.yml -u root -i marathon,.

Do a docker run --rm -p 80:80 nginx, and open http://marathon and see the Nginx default message. It works!

---

We are seeing repeating patterns, so let's make some bash functions to save us the trouble of re-typing:

function play() {
    ansible-playbook -vvv $1 -u root -i marathon,
}

Works well.

Developing the app

For today, I want to start with something simple. Let's write some web scraping scripts with Python and display the aggregated data in a nice formated spread sheet so that we can sort it.

I have an idea: How about we archive all the DevOps jobs on Indeed (Canada), and put them in a spreadsheet.

Why does anybody want this?

It's much easier to sort with a spreadsheet than clicking the filtering criterias on the website. Plus, I can still keep a record of the jobs that are no longer available. So that we can run some statistics on the job market history.

Let's get started!

---

First, we need to figure out what to search for. Let's play with the advanced search feature.

We want all the jobs in Canada so we will leave out the location; the age of the post will be set to "since yesterday", because our script will run daily. Let's display 50 results per page.

This is the url to run:

It gives me 91 results on 2 pages.

We can see that the search gives some false positive. An "Advanced Web UI Application Developer" cannot be a Devops job just because it has "Nice to have: Devops Experience" in it. Let's match the title but not the description. We will match "devops" as well as "Development Operations", since it's a word game.

After some trail and error, I find this url gives the best result:

It will give jobs for "Devops" or "Development Operation", case ignored.

---

Then, we need to figure out what information we need to scrap.

We will scrap:

1. Job Title
2. Company name
3. Short Description
4. Posting time (we can use yesterday as timestamp, as long as we run the script at 00:01 every day.)
5. Location
6. salary
7. Boolean: can be applied on indeed.

I will try requests-html Python library by Kenneth Reitz. It looks really easy to use, and it supports Javascript.

Run pipenv install requests-html. Then pipenv shell. Then python.

With the help of another tool called SelectorGadget, I can easily find the CSS selectors I need.

It went smoothly, first draft took about 45 minutes. Here's the code:

from requests_html import HTMLSession
import datetime

def get_text(i, sel):
    try:
        return i.find(sel)[0].text
    except Exception:
        return ""

s = HTMLSession()
r = s.get('https://ca.indeed.com/jobs?as_and=&as_phr=&as_any=&as_not=&as_ttl=Devops+or+%22Development+Operation%22&as_cmp=&jt=all&st=&salary=&radius=50&l=&fromage=1&limit=50&sort=date&psf=advsrch')

r.html.render()
l = r.html.find('.clickcard')

res = []

for i in l:
    d = {}
    d['ttl'] = get_text(i, '.jobtitle .turnstileLink')
    d['comp'] = get_text(i, '.company')
    d['loc'] = get_text(i, '.location')
    d['desc'] = get_text(i, '.summary')
    d['apply'] = i.find('.iaLabel') != []
    d['date'] = datetime.datetime.now().strftime("%Y-%m-%d")
    d['sal'] = get_text(i, '#resultsCol .no-wrap')
    res.append(d)

import pprint; pprint.pprint(res)

---

I decided to use Google Spreadsheet to store and display the data. Following this tutorial.

I manually created a spreadsheet on my Google account. And shared with the API user. Now I just need Python to insert the records in the spreadsheet.

I also added the first row as header row; debugging the selector for salary; and write deduplication feature in Python code. Now it works very well.

Visit the Google Sheet!

---

Now I will package the code in a docker image and pull it from the server, which will run a docker container daily, with a cron job.

Build the image:

FROM ubuntu:xenial

RUN apt-get -y update && apt-get install -y python3 python3-pip
RUN pip install -r requirements.txt

CMD python insert_gsheet.py

It took me 6 hours 20 minutes straight working on this thing. And I cannot deploy it with Docker. It seems that even with Docker, we are still at works on my laptop but not on prod stage. I'm blocked at this issue: pyppeteer.errors.BrowserError: Failed to connect to browser port: http://127.0.0.1:45439/json/version, which seems to be related to docker. Might be related here.

Finally, I got it working after following the instructions here. I got one giant Dockerfile.

---

Now we push the image to Docker Hub: docker tag dohsimpson:indeed_jobs dohsimpson/indeed_jobs docker push dohsimpson/indeed_jobs. Then pull from the server: docker pull dohsimpson/indeed_jobs. Push takes forever because my slow uplink.

Lastly, we add a cron entry with Ansible.

- hosts: all
  tasks:
  - name: Run indeed_jobs every day after midnight
    cron:
      name: Run indeed jobs docker container
      job: docker run --rm dohsimpson/indeed_jobs
      minute: "1"
      hour: "0"

All seems working!

Lesson learned

Do not use Docker if you are working solo, not worth the trouble.

TODO:

• The script doesn't handle pagination, but if we run it frequent enough, it won't be a problem.
• I will leave out any CSS magic for now.
• Try package the app with Dokku, it might be easier.