Automatic TeslaCam And Sentry Mode Video Processing In Azure – Part 1: The Overview

TLDR: With this solution, you will get a notification on your phone with direct links to the TeslaCam and Sentry Mode footage of that day, after you arrive home and the automatic processing of the videos is completed. To save time reviewing the videos, the 4 different video angles of each event (front, back, left, right camera) are combined into a single video! As a bonus, all videos are archived on Azure Storage. This is part 1 of a series of blog posts about this solution, with an introductory overview, an explanation of the ‘why’, and the prerequisites.

Reviewing Tesla camera footage.. a cumbersome job!

Tesla car owners that store the car’s dashcam and sentry mode videos on a thumb drive (or SSD) probably recognize it’s a lot of work to review those videos to see what was going on and identify suspicious behavior or situations. And especially sentry mode can create a lot of videos, with almost all of them non-eventfull (caused by an animal walking by, raindrops, traffic passing by, etc.). And the fact that each event saves 4 different files (front, back, left, right camera) means it is simply a lot of work to review all those videos. I haven’t met any Tesla car owner that does that.

Even if you have one of those USB thumb-drives that you can plug in your phone, with a nice app to review the videos, it is too cumbersome and you will probably only review the videos when you have a feeling – or know for sure – there is something on those videos you want to check.

The best of both worlds: USB thumb drive & mini pc

To make this process a little less cumbersome, a clever mind1 had an idea to use a RaspberryPi Zero W that can act as both a USB drive and a computer at the same time, automatically uploading the TeslaCam videos to a file server in your home network when you come home. This became the GitHub project called TeslaUSB. This prevents you from having to remove the drive from the car and plugging it in your PC/laptop or phone. After the software recognizes your home WiFi2 when you come home, it will automatically start copying the videos.

Initially, I started out with a setup where the footage was automatically archived to the RaspberryPi hosting my Home Automation solution, but quite soon I ran out of storage and the review process was still taking way too much time. The TeslaUSB project also supports archiving the videos to cloud services like Azure Blob Storage, so that’s what I’m doing now: after coming home, the RaspberryPi automatically uploads all videos of that day to Azure. Not a single user action required.

Big time saver: combining 4 camera angles into 1 video

I also stumbled upon a project on GitHub that combines the videos of the 4 different angles into a single video, and boy this is handy. Usually only one or two of the cameras have something interesting per event. So this is a big time saver. Check out the still below, that I took of a recent video. The project can handle more layouts, but I think this one is the best:

Example merged TeslaCam video

Another big feature of this project/script is that it’s capable of identifying motion in the video, and it can automatically remove frames without motion. One recent video of about 10 minutes got reduced to only this short video (this example is a gif looping):

I assumed (and still do) that the RaspberryPi Zero W in the car, hosting the TeslaUSB software, does not have enough compute power to execute this script, so I got creative and found a perfect service to perform this task: Azure Batch. I didn’t have any experience with this service, but that was a challenge I gladly accepted.

The notification

The end result of the processing steps (for now) is a notification on my phone, telling me a new video is ready to be reviewed:

That’s awesome, but it gets even better. I can open the notification, and it serves a direct link to the video:

The solution architecture

Combining everything into one overview you get the diagram below.
I’ve created this using CloudSkew. A fairly new project that lets you create Azure diagrams for free. Go try it out! You can also find the source on GitHub and contribute.

Daves Automatic TeslaCam video procesing solution architecture

Prerequisites

The simple components of the solution: the hardware!

  • MicroSD card (minimal size: 16GB)
  • RaspberryPi W Zero + optional: a nice case
  • USB+data cable to connect the RaspberryPi to the USB outlet in the car
  • Card reader for your MicroSD card (to flash it from your PC)
  • Optional: a nice hub/dock to hide the USB cables and RaspberryPi, I’ve ordered the one from taptes.com but I have not received it yet.

The ‘software’ prerequisites you need for this solution:

  • An Azure subscription
  • A Power Automate environment, and the app installed on your phone
  • An account in IFTTT, and the app installed on your phone

To be continued…!

This was part 1 with an overview of the solution. If you want to create a solution like this yourself, get those prerequisites in place and watch this blog for the next posts in this series where we configure all the services and connect them together.

Looking further ahead

While I’m still finetuning the current solution here and there, I would very much like to extend it to receive near-real-time notifications of Sentry Mode events, with the link to the video of course. I hope Tesla will make this available, but until that moment I am considering options to make this possible myself.

Other development options I have in mind:

  • Enhance object and face detection in the footage
  • Add a Power App for management of the event videos, with a historical catalog of all videos. And a way to quickly scan multiple events (e.g. videos) one after another, which right now is only possible by opening the notification in the Power Automate app and the navigation for that is very clumsy. Probably I’ll also tie the notifications to the app, deep linking to the app and viewing the video from within the app.

1 The first TeslaUSB project (FYI: I’m not using that one) on GitHub was initially created by user cimryan with permissions of the original author of the first scripts, as you can read here on Reddit. Later on, GitHub user marcone created a fork that is now a very active project, and I’m using that fork!
2 Your car needs to be parked close enough to your home so the Raspberry Pi can connect to your home WiFi.

Pass the sauce
More from Dave Ruijter

The Absolute Guide to Selecting Between Azure Analysis Services and Power BI Premium

It’s one of the questions I get asked most often these days....
Read More

6 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *