Automatic TeslaCam And Sentry Mode Video Processing In Azure – Part 1: The Overview

TLDR: With this solution, you will get a notification on your phone with direct links to the TeslaCam and Sentry Mode footage of that day, after you arrive home and the automatic processing of the videos is completed. To save time reviewing the videos, the 4 different video angles of each event (front, back, left, right camera) are combined into a single video! As a bonus, all videos are archived on Azure Storage. This is part 1 of a series of blog posts about this solution, with an introductory overview, an explanation of the ‘why’, and the prerequisites.

Reviewing Tesla camera footage.. a cumbersome job!

Tesla car owners that store the car’s dashcam and sentry mode videos on a thumb drive (or SSD) probably recognize it’s a lot of work to review those videos to see what was going on and identify suspicious behavior or situations. And especially sentry mode can create a lot of videos, with almost all of them non-eventfull (caused by an animal walking by, raindrops, traffic passing by, etc.). And the fact that each event saves 4 different files (front, back, left, right camera) means it is simply a lot of work to review all those videos. I haven’t met any Tesla car owner that does that.

Even if you have one of those USB thumb-drives that you can plug in your phone, with a nice app to review the videos, it is too cumbersome and you will probably only review the videos when you have a feeling – or know for sure – there is something on those videos you want to check.

The best of both worlds: USB thumb drive & mini pc

To make this process a little less cumbersome, a clever mind1 had an idea to use a RaspberryPi Zero W that can act as both a USB drive and a computer at the same time, automatically uploading the TeslaCam videos to a file server in your home network when you come home. This became the GitHub project called TeslaUSB. This prevents you from having to remove the drive from the car and plugging it in your PC/laptop or phone. After the software recognizes your home WiFi2 when you come home, it will automatically start copying the videos.

Initially, I started out with a setup where the footage was automatically archived to the RaspberryPi hosting my Home Automation solution, but quite soon I ran out of storage and the review process was still taking way too much time. The TeslaUSB project also supports archiving the videos to cloud services like Azure Blob Storage, so that’s what I’m doing now: after coming home, the RaspberryPi automatically uploads all videos of that day to Azure. Not a single user action required.

Big time saver: combining 4 camera angles into 1 video

I also stumbled upon a project on GitHub that combines the videos of the 4 different angles into a single video, and boy this is handy. Usually only one or two of the cameras have something interesting per event. So this is a big time saver. Check out the still below, that I took of a recent video. The project can handle more layouts, but I think this one is the best:

Example merged TeslaCam video

Another big feature of this project/script is that it’s capable of identifying motion in the video, and it can automatically remove frames without motion. One recent video of about 10 minutes got reduced to only this short video (this example is a gif looping):

I assumed (and still do) that the RaspberryPi Zero W in the car, hosting the TeslaUSB software, does not have enough compute power to execute this script, so I got creative and found a perfect service to perform this task: Azure Batch. I didn’t have any experience with this service, but that was a challenge I gladly accepted.

The notification

The end result of the processing steps (for now) is a notification on my phone, telling me a new video is ready to be reviewed:

That’s awesome, but it gets even better. I can open the notification, and it serves a direct link to the video:

The solution architecture

Combining everything into one overview you get the diagram below.
I’ve created this using CloudSkew. A fairly new project that lets you create Azure diagrams for free. Go try it out! You can also find the source on GitHub and contribute.

Daves Automatic TeslaCam video procesing solution architecture

Prerequisites

The simple components of the solution: the hardware!

  • MicroSD card (minimal size: 16GB)
  • RaspberryPi W Zero + optional: a nice case
  • USB+data cable to connect the RaspberryPi to the USB outlet in the car
  • Card reader for your MicroSD card (to flash it from your PC)
  • Optional: a nice hub/dock to hide the USB cables and RaspberryPi, I’ve ordered the one from taptes.com but I have not received it yet.

The ‘software’ prerequisites you need for this solution:

  • An Azure subscription
  • A Power Automate environment, and the app installed on your phone
  • An account in IFTTT, and the app installed on your phone

To be continued…!

This was part 1 with an overview of the solution. If you want to create a solution like this yourself, get those prerequisites in place and watch this blog for the next posts in this series where we configure all the services and connect them together.

Looking further ahead

While I’m still finetuning the current solution here and there, I would very much like to extend it to receive near-real-time notifications of Sentry Mode events, with the link to the video of course. I hope Tesla will make this available, but until that moment I am considering options to make this possible myself.

Other development options I have in mind:

  • Enhance object and face detection in the footage
  • Add a Power App for management of the event videos, with a historical catalog of all videos. And a way to quickly scan multiple events (e.g. videos) one after another, which right now is only possible by opening the notification in the Power Automate app and the navigation for that is very clumsy. Probably I’ll also tie the notifications to the app, deep linking to the app and viewing the video from within the app.

1 The first TeslaUSB project (FYI: I’m not using that one) on GitHub was initially created by user cimryan with permissions of the original author of the first scripts, as you can read here on Reddit. Later on, GitHub user marcone created a fork that is now a very active project, and I’m using that fork!
2 Your car needs to be parked close enough to your home so the Raspberry Pi can connect to your home WiFi.

Pass the sauce

This Post Has 12 Comments

  1. Pieter van Maasdam

    Very cool Dave!

  2. Joe

    This looks like a wonderful tool! What sort of minimal costs are we looking at for the Azure and power automate requirements?

    1. Dave Ruijter

      Hi Joe,

      Hard to say exactly what the monthly cost would be. You can use the Azure cost calculator to make your own estimation: https://azure.microsoft.com/en-us/pricing/calculator/.
      It will be about the number of hours you run the low-priority machines per month (a couple of dollars in my case), and the total amount of data in the storage account (also a couple of dollars in my case).
      For the storage account, it is important to keep an eye on the archive folder, as that will continue to grow if you don’t delete old data.

  3. David

    Hi Joe,

    Awaiting my M3 and have been turning a few ideas in my head around as I’m not sure I’m competent enough to execute them but I was sure someone had tried. Great work.

    A few questions theoretical questions if you don’t mind:

    1) Would it be possible to have a mifi drive in the car connected to a power bank which in turn was plugged in to power the arm rest (This would provide a 24/7 wifi signal.

    2) Could the rasberry pi then transmit any data written to the saved clips folder once written that way there is an almost immediate backup of any intrusion attempt?

    1. Dave Ruijter

      Hi David,

      Great ideas! The 24/7 wifi signal would work I think.
      But, the rpi immediately copying the videos would need some changes in the setup/code, as currently the scripts monitor for a WiFi reconnect (e.g. coming home).

      Dave

      1. David

        Hi Dave,

        That’s good to hear. Our parking is in a shared parking with no chance of our home wifi getting there which is why I’m considering having a dedicated one in the car. The wifi in the vehicle would also help with updates overnight.

        The only bit left would be for the rpi to copy to cloud once there was a change in the sentrycam folder.

        This is not necessary but in the event a burglar is familiar with the fact that footage is being stored locally and considers removing the drive there would be a backup online.

        Thanks again.
        David

        1. Dave Ruijter

          Yeah I dig the idea! I’d like to have this as well. Getting a notification when sentry is triggered, and being able to immediately see the footage is very valuable. I think I’ve seen an issue/request posted on this in the teslausb Git repo.. Have you searched there?

          Ps what kind of device are you thinking of using for the access point? And probably it would require a SIM card? My cell provider charges €5/month for a second SKM πŸ˜•

  4. Paul S.

    Can you use those Wi-Fi SD cards so when you get home and connect to your wifi, You can access the SD card and copy the data? probably can automate

Leave a Reply