2012-06-13

Motion Google Drive Uploader and Emailer

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0



I'm using the brilliant Motion software with my Raspberry Pi to capture goings-on around my house. This blog post: Battery powered, Wireless, Motion detecting Raspberry Pi and this blog post: Raspberry Pi Webcam explain how.

I set up Lighttpd to point at the directory where Motion was creating video files so that I could go and check periodically to see if any new events had been captured. This works well - I just set up my home router so I could access the http over the Internet and then download the AVIs to my laptop or whatever. The down side to this is that you have to pro-actively look to see if there are new videos and secondly it puts additional strain on the Raspberry Pi to host the files for repeated download.

So, I have created a Python script to upload the AVI to Google Drive, get a URL to view the video on line and then email this using the same GMail account to an address. To keep the Raspberry Pi tidy the script then deletes the local file.

If you have the Google Drive Sync application running too you can see the files as they arrive which is fun:

Source


#!/usr/bin/python2
'''
Created on 6 Jun 2012

@author: Jeremy Blythe

Motion Uploader - uploads videos to Google Drive

Read the blog entry at http://jeremyblythe.blogspot.com for more information
'''

import smtplib
from datetime import datetime

import os.path
import sys

import gdata.data
import gdata.docs.data
import gdata.docs.client
import ConfigParser

class MotionUploader:
    def __init__(self, config_file_path):
        # Load config
        config = ConfigParser.ConfigParser()
        config.read(config_file_path)
        
        # GMail account credentials
        self.username = config.get('gmail', 'user')
        self.password = config.get('gmail', 'password')
        self.from_name = config.get('gmail', 'name')
        self.sender = config.get('gmail', 'sender')
        
        # Recipient email address (could be same as from_addr)
        self.recipient = config.get('gmail', 'recipient')        
        
        # Subject line for email
        self.subject = config.get('gmail', 'subject')
        
        # First line of email message
        self.message = config.get('gmail', 'message')
                
        # Folder (or collection) in Docs where you want the videos to go
        self.folder = config.get('docs', 'folder')
        
        # Options
        self.delete_after_upload = config.getboolean('options', 'delete-after-upload')
        self.send_email = config.getboolean('options', 'send-email')
        
        self._create_gdata_client()

    def _create_gdata_client(self):
        """Create a Documents List Client."""
        self.client = gdata.docs.client.DocsClient(source='motion_uploader')
        self.client.http_client.debug = False
        self.client.client_login(self.username, self.password, service=self.client.auth_service, source=self.client.source)
               
    def _get_folder_resource(self):
        """Find and return the resource whose title matches the given folder."""
        col = None
        for resource in self.client.GetAllResources(uri='/feeds/default/private/full/-/folder'):
            if resource.title.text == self.folder:
                col = resource
                break    
        return col
    
    def _send_email(self,msg):
        '''Send an email using the GMail account.'''
        senddate=datetime.strftime(datetime.now(), '%Y-%m-%d')
        m="Date: %s\r\nFrom: %s <%s>\r\nTo: %s\r\nSubject: %s\r\nX-Mailer: My-Mail\r\n\r\n" % (senddate, self.from_name, self.sender, self.recipient, self.subject)
        server = smtplib.SMTP('smtp.gmail.com:587')
        server.starttls()
        server.login(self.username, self.password)
        server.sendmail(self.sender, self.recipient, m+msg)
        server.quit()    

    def _upload(self, video_file_path, folder_resource):
        '''Upload the video and return the doc'''
        doc = gdata.docs.data.Resource(type='video', title=os.path.basename(video_file_path))
        media = gdata.data.MediaSource()
        media.SetFileHandle(video_file_path, 'video/avi')
        doc = self.client.CreateResource(doc, media=media, collection=folder_resource)
        return doc
    
    def upload_video(self, video_file_path):
        """Upload a video to the specified folder. Then optionally send an email and optionally delete the local file."""
        folder_resource = self._get_folder_resource()
        if not folder_resource:
            raise Exception('Could not find the %s folder' % self.folder)

        doc = self._upload(video_file_path, folder_resource)
                      
        if self.send_email:
            video_link = None
            for link in doc.link:
                if 'video.google.com' in link.href:
                    video_link = link.href
                    break
            # Send an email with the link if found
            msg = self.message
            if video_link:
                msg += '\n\n' + video_link                
            self._send_email(msg)    

        if self.delete_after_upload:
            os.remove(video_file_path)

if __name__ == '__main__':         
    try:
        if len(sys.argv) < 3:
            exit('Motion Uploader - uploads videos to Google Drive\n   by Jeremy Blythe (http://jeremyblythe.blogspot.com)\n\n   Usage: uploader.py {config-file-path} {video-file-path}')
        cfg_path = sys.argv[1]
        vid_path = sys.argv[2]    
        if not os.path.exists(cfg_path):
            exit('Config file does not exist [%s]' % cfg_path)    
        if not os.path.exists(vid_path):
            exit('Video file does not exist [%s]' % vid_path)    
        MotionUploader(cfg_path).upload_video(vid_path)        
    except gdata.client.BadAuthentication:
        exit('Invalid user credentials given.')
    except gdata.client.Error:
        exit('Login Error')
    except Exception as e:
        exit('Error: [%s]' % e)


Installation

If you haven't already got Python and Pip installed then do it now. On my Arch Linux Raspberry Pi I did this:
pacman -S python2
pacman -S python2-pip
Note: This script has been tested on Python 2.7.3
Next get the Google gdata library:
pip2 install gdata
Now download the two files above to somewhere on your machine. Make uploader.py executable:
chmod +x uploader.py
Open uploader.cfg to enter your settings:
[gmail]
# GMail account credentials
name = My Name
user = gmailusername
password = gmailpassword
sender = me@gmail.com

# Recipient email address (could be same as from_addr)
recipient = me@gmail.com

# Subject line for email
subject = Motion detected

# First line of email message
message = Video uploaded

[docs]
# Folder (or collection) in Docs where you want the videos to go
folder = motion

[options]
# Delete the local video file after the upload
delete-after-upload = true

# Send an email after the upload
send-email = true
Make sure you create a folder in your Google Drive to match the setting in the uploader.cfg file. This is the folder where the videos will go.
At this point it's worth testing the script e.g.
./uploader.py /etc/motion/uploader.cfg /srv/http/motion/cam1/85-20120608194940.avi
If all's well you can now change the Motion setting like so:
on_movie_end /root/py/uploader.py /etc/motion/uploader.cfg %f

Enjoy!

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0


37 comments:

Dean said...

Amazing articles! I have been following your installation of the webcam and this has been a great reference for info for the Raspberry Pi!

Anonymous said...

nice one! thanks a lot :)
I try to make use of your script on an ubuntu box, but after installing python-gdata, i have an error on importing gdata.data?
no module named data...
I have to fix that

whocares said...

Fantastic stuff and worked a treat, I love it.

Many thanks for the amount the work that's clearly gone into this and that you chose to share it.

Very much appreciated

thehill06 said...

Jeremy, many thanks for putting these Pi tutorials together!

I've got Motion up and running and am now trying to get the GDrive script to upload the video/images. When I run the script I get the generic "Login Error" and I'm not sure how to troubleshoot further.

I've done a bit of troubleshooting up to this point, including pointing the script to a non-existent folder on gdrive, which led to me getting the "Could not find the %s folder" response. To me, this rules out a log-in problem.

Any ideas?

jerbly said...

@thehill06 looks like you might have to dig deeper into the gdata libraries to work out what's going on here - I wonder if there's more info behind that gdata.client.Error exception? In these situations I would pepper the code with print statements to see where it gets to before it falls over to hone in on the problem. Good Luck!

thehill06 said...

Got it working! This is brilliant. Thanks for your work and I look forward to seeing your other Pi projects.

jerbly said...

@thehill06 Great! Check out my new post that hooks up LEDs and buttons to Motion: Raspberry Pi GPIO and Motion

Xen said...

@thehill06, I've got the same problem with "login error", however I'm not too sure where to start to debug the problem. What did you do to resolve the issue?

thehill06 said...

@ Xen, it turns out when I was doing my test upload I was trying to upload an image rather than a video file. Once I pointed the script to a video the issue was resolved.

Anonymous said...

Tanks for that excellent work! But i still have a litle problem...

When i execute the python script, i get an error:

Error: ['DocsClient' object has no attribute 'GetAllResources']

Can anyone help me?

Greetings from germany phate127

Anonymous said...

Does anyone have an example of how to upload an image instead of the avi to google.
I've managed to get it to upload a jpeg by changing the doc type to image/jpeg, but viewing in google docs just opens up a blank document.

Lewis said...

Hi Jeremy, thanks for the post - using this for my homebrew burglar detection system! A quick question - did you install a desktop environment and do your editing from a text editor or did you just use nano to edit the python scripts from the terminal?

jerbly said...

@lewis I just use nano. I always connect in over ssh using PuTTY to do everything on the pi. It means less to install and more resources available. I also use svn to keep the code safe and then I can write it on my laptop and update on the pi.

Lewis said...

Thanks for the quick response Jeremy - that clarifies it for me! Great tutorials.

Magic919 said...

Tanks for that excellent work! But i still have a litle problem...

When i execute the python script, i get an error:

Error: ['DocsClient' object has no attribute 'GetAllResources']

Can anyone help me?

Greetings from germany phate127

Try using the later version of gdata. I got this error with 2.0.14, but 2.0.17 solved it.

A.J.Bevan said...

Very nice work Jeremy. I have completed most of this setup, except the video upload. When testing the script as you describe it works well. However when adding into motion.conf I see that we are not specifying the full path and video name to upload. Filename I can understand, but where does your script get the path to where motion stores the video files?
I see the variable "video_file_path" but dont see where it is defined.

Many Thanks.

A.J.Bevan said...

Jeremy, please ignore my previous comment, have fixed it - IT WORKS !!

Many thanks for your efforts.

CValdesS said...

Hi Jeremy

Just got Error: ['DocsClient' object has no attribute 'GetAllResources']

ubuntu server 12.04 LTS
python 2.7.3-0ubuntu2
python-gdata 2.0.14-2

any solution??

thanks

mohsin said...

Hi Jeremy,
Thanks for the great post. I am got everything running but stuck on the last step. I can send force the upoloader to copy the video manually.

But it does not work when the motion is running. It only uploads the vipdeo when I stop the motion by pressing control+c

Any clues ?

Thanks
Mohsin

mohsin said...

Also I only get the following trace when I press control+c after which it sends the file.

Executing external command '/home/mohsin/Downloads/uploader.py /home/mohsin/Downloads/uploader.conf /tmp/motion/01-20120909040044.avi

And I dont see this kind of trace while the motion is running.

Thanks,
Mohsin

mohsin said...

Ignore my earlier comments. It does take a bit of time to send the message when the motion is running. It actually doe send the message.

jerbly said...

@cvaldess - I haven't seen that error before but I suspect it is something to do with the gdata install. You could try doing some other basic gdata things from the google tutorials to see if it's working.

CValdesS said...

Jeremy, run some examples from gdata-docs and work ok, think they don't support GetAllResources any more.. but not a python programer. thxn

Kenny Millington said...

Hi,

Thanks for the script/blog post(s) - all very useful.

Just wanted to highlight a minor-minor bug (sorry!) that might help other people...

self.client.client_login(self.sender, ...

should be

self.client.client_login(self.username, ...

That way it will log in with the username and not the sender e-mail address (in the case they are different) - that had me for a good while trying to figure out if Google's two-factor auth was to blame (I was using an app-specific password).


Many thanks,
Kenny

ddriver said...

Hi, thank you for the great articles, I've successfully managed to get motion working on a fixed IP address and get to a point where i have the two scripts in directories on my RPi.

I'm however having trouble with running and testing the 'uploader.py' script. I keep getting an error saying '-bash ./uploader.py: No such file or directory'

I've only been playing with the Linux shell for a week and can't say I'm finding it as easy as I would have liked, so any help would be much appreciated!

Thanks

Anonymous said...

Great script, Thanks!

But could it be made to upload JPEG images instead? I've changed the MIME and type="images", however it seems Google tries to convert the image on upload. The URL needs to have '?convert=false' added at the end when uploading, but I can't figure out how. Any help?

southscanner said...

All working very well. I modified to upload jpegs instead of avi's:

doc = gdata.docs.data.Resource(type='image', title=os.path.basename(video_file_path))
media = gdata.data.MediaSource()
media.SetFileHandle(video_file_path, 'image/jpeg')


One question though, I then thought I'd add the URL to the mail but the links don't work. They begin

https://docs.google.com/feeds/default/contentmedia/folder

and when accessed I get Invalid Request URI.

Any ideas?

Anonymous said...

My jpegs are still coming in at 1kb, and as a .gdoc document. I can't figure out how to add the "?convert=false"

mroth said...
This comment has been removed by the author.
mroth said...

To add the ?convert=false while still posting to a collection, make @southscanner's changes, then
replace

doc = self.client.CreateResource(doc, media=media, collection=folder_resource)

with

create_uri = folder_resource.get_resumable_create_media_link().href + '?convert=false'
doc = self.client.CreateResource(doc, create_uri=create_uri, media=media)

See
http://packages.python.org/gdata/docs/api.html
--and--
http://planzero.org/blog/2012/04/13/uploading_any_file_to_google_docs_with_python

Anonymous said...

Thanks for this script!

The uploader works when running the script manually but not when the video has finished recording. I am using the exact same paths in both, except instead of ending with the filename in the config file I end with %f

Any ideas?

Patrick said...

Thanks for the scrip.I have the same problem as my previous poster, please help.

Unknown said...

Very good!! Thanks!

Derek Bolthausen said...

Not just for Pi lol. What a great post! Thanks Jeremy as this has helped me set up my motion setup with 4 webcams sending just the ones I need to Google Drive in photos. Great Post !

Anonymous said...

for wheezy:
python not python2
pip not pip2
change first line of script (interpreter) to python2.7
and works like a charm

Anonymous said...

I am using the ip camera to ftp the images to pi then upload it to drive. I can manually upload the images one by one but not when I use motion.cfg. I got both scripts and the jpgs are in one directory.
SK

Tricia D said...

Hi, I have never used python or linux before and was wondering if you could explain exactly where to plug in the filenames/paths. I have tried a millions combos and am very frustrated.

any help is greatly appreciated.