Question

I have pieced together the following script to generate a daily report on a remote Asterisk / Vicidial server. The script grabs the source from the report, performs some formatting, saves the result as a text file and then sends it via smtp to my boss for review. I’m currently running the script on a cron job locally and it works perfectly, however I would like to be able to run it on my VPS; problem being, the script pops a Firefox window and the VPS does not have X or any sort of GUI, and consequently, Firefox will not open, and the data cannot be retrieved.

Just to clarify, I have Firefox and all required modules installed and the VPS is essentially identical to my local machine, bar X and GUI (Debian Lenny).

If some someone could provide any sort of help on how to modify this script in order to work without X / GUI, it would be greatly appreciated!

Thanks, Toby.

import contextlib
import selenium.webdriver as webdriver
import lxml.html as LH
import lxml.html.clean as clean
import csv 
import sys
import smtplib
from email.mime.text import MIMEText
import email.mime.application
import email
import mimetypes
import datetime

date=datetime.date.today()

url="http://myuser:mypass@ipaddress"+ str(date) + "some_other_string"

ignore_tags=('script','noscript','style')
with contextlib.closing(webdriver.Firefox()) as browser:
    browser.get(url)
    content=browser.page_source
    cleaner=clean.Cleaner()
    content=cleaner.clean_html(content)    
    with open('vicidial_data.html','w') as f:
       f.write(content.encode('utf-8'))
    doc=LH.fromstring(content)
    with open('grab_raw.txt','w') as f:
        for elt in doc.iterdescendants():
            if elt.tag in ignore_tags: continue
            text=elt.text or ''
            tail=elt.tail or ''
            words=' '.join((text,tail)).strip()
            if words:
                words=words.encode('utf-8')
                f.write(words+'\n') 

grab=open( 'grab_raw.txt', 'r' )
grab_list=grab.readlines()
grab.close()
del grab_list[0:21]
grab_out=open("Vicidial_Report-"+str(date)+".txt", 'w')
grab_out.writelines(grab_list)
grab_out.close()

msg=email.mime.Multipart.MIMEMultipart()
msg['Subject']='Vicidial call-report' 
msg['From']='[e-mail]'
msg['To']='[e-mail]'

body = email.mime.Text.MIMEText("Please find attached call-report for " + str(date))
msg.attach(body)

filename= "Vicidial_Report-"+str(date)+".txt"
fp=open(filename,'rb')
att = email.mime.application.MIMEApplication(fp.read(),_subtype="text")
fp.close()
att.add_header('Content-Disposition','attachment',filename=filename)
msg.attach(att)

s = smtplib.SMTP('smtp.gmail.com')
s.starttls()
s.login('[e-mail]','mypassword')
s.sendmail('[e-mail]',['[e-mail]', '[e-mail]', ],     msg.as_string())
s.quit()

Answer 1

Corey Goldberg explains how to do this using pyvirtualdisplay.

You’ll need to be able to install pyvirtualdisplay, xvfb, and xserver-xephyr, however.

import contextlib
import selenium.webdriver as webdriver
display = pyvirtualdisplay.Display(visible = False, size = (800, 600))
display.start()
with contextlib.closing(webdriver.Firefox()) as driver:    
    driver.get('http://www.google.com')
    print driver.title
    # Google
display.stop()

On Ubuntu/Debian the necessary packages can be installed with

sudo apt-get install python-setuptools
sudo apt-get install xvfb
sudo apt-get install xserver-xephyr
sudo easy_install pyvirtualdisplay