Comma AI - Video Processing Driving Data

Author

Collin Real

Introduction

      Comma.ai is one of the few respectable tech companies offering one of the most advanced self-driving products: the comma 3x. - Automates ~70+% of daily driving. - Performs exceptionally well on highways and other roads with identifiable lanes. - Installed and mounted on a car’s front windshield, so it can receive a live data feed of the road. - Using this live feed, the comma 3x projects the path for the vehicle to follow.

Comma API

      Comma uploads driving data to its servers to train better models and improve the self-driving experience over time. We can access our driving data using the comma API. Using our driving data, we can create metrics to analyze our driving patterns and behavior.

My Comma 3x device

Visit the website for a more comprehensive overview: comma.ai

Set Up Virtual Environment/Install Dependencies (Mac)

Execute these commands in your terminal

  • Create local virtual env: python3 -m venv .venv
  • Activate local virtual env: source .venv/bin/activate
  • Install Python dependencies: pip3 install -r requirements.txt
  • Install Homebrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  • Install ‘ffmpeg’ via Homebrew: brew install ffmpeg
  • OPTIONAL - Connect your personal Comma AI device:
    • touch .env
    • nano .env - opens .env file in terminal
    • COMMA_AI_KEY="insert your Comma API key"
    • DONGLE_ID="insert your dongle ID"
    • Save file and exit nano

Import Libraries & Set Configurations

import pandas as pd # data processing
import urllib.request # download file from URL
import ssl # bypass SSL certificate
import warnings # ignore non-critical warning outputs
import cv2 # video processing
import matplotlib.pyplot as plt # data visualization
import matplotlib.image as mpimg # data visualization
import subprocess # running terminal commands in Python script
import seaborn_image as isns # data visualization
from requests import get # API request
from time import sleep # Prevent triggering the API limit
from os import environ, listdir, mkdir, makedirs # directory manipulation & file saving
import os.path
from dotenv import load_dotenv # load environment variables
from tqdm import tqdm # added as a meme, prints unnecessary loading bar in terminal during for loops
from moviepy.editor import VideoFileClip, concatenate_videoclips # video editing
plt.style.use('ggplot')
warnings.filterwarnings('ignore')
ssl._create_default_https_context = ssl._create_unverified_context
pd.set_option('display.max_columns', None)
load_dotenv()
True

Create Variables for API Requests

The first step to receive the recording of my longest trip (College Station) since installation is making sure I send the correct parameters to the API endpoint. Comma.ai’s API requests require an authentication token and the dongle ID of a user’s Comma device.

TOKEN= environ.get('COMMA_AI_KEY')
DONGLE_ID = environ.get('DONGLE_ID')
headers = {
    'Authorization': 'JWT {}'.format(TOKEN)
}
BASE_URL = 'https://api.commadotai.com'

Create/Check File Paths Exist

route_data_path = 'data/route-data'
vid_urls_path = 'data/vid-urls'
vid_save_files_path = 'data/vid-files'
mp4_directory = 'data/vid-mp4'
full_vid_path = 'data/vid-full'
images_path = 'data/route-images'

paths = [
    route_data_path,
    vid_urls_path,
    vid_save_files_path,
    mp4_directory,
    full_vid_path,
    images_path,
]

for route_vid_path in paths:
    if os.path.exists(route_vid_path) == False: mkdir(route_vid_path)
    else: pass

API Request #1 - Returns User Driving Data

After creating the API variables, we can request the API endpoint which returns our driving data in the response output. The first API request will return various metrics from all of my driving trips since installing my Comma 3x. It will provide us with the route name for every trip. For our current task, we have chosen our longest trip by miles, so we will sort the dataset by longest trip to identify the route name. After sorting by descending order, the first row’s value in column fullname is the route name.

def query_route_data(BASE_URL: str):
    # Send API request
    resp = get(
        f'{BASE_URL}/v1/devices/{DONGLE_ID}/routes_segments?start=1706050612200&end=1811678741855', headers=headers, 
        verify=False)

    # Convert API response to JSON
    content = resp.json()

    # Create DataFrame w/ API Response
    df = pd.DataFrame(content)

    # Remove latitude, longitude variables for privacy.
    df = df[[
        'fullname', 'length', 'create_time', 'end_time_utc_millis',
        'end_time', 'init_logmonotime', 'maxqcamera', 'maxqlog', 
        'platform', 'procqcamera', 'procqlog', 'segment_end_times', 
        'segment_numbers', 'segment_start_times', 'start_time_utc_millis', 'version'
    ]]

    # Time metric conversions
    df['time_diff_millis'] = df['end_time_utc_millis'] - df['start_time_utc_millis']
    df['time_diff_seconds'] = df['time_diff_millis'].__truediv__(1000)
    df['time_diff_minutes'] = df['time_diff_seconds'].__truediv__(60)
    df['time_diff_hours'] = df['time_diff_minutes'].__truediv__(60)
    df['end_time'] = pd.to_datetime(df['end_time']).dt.strftime("%Y-%m-%d")

    # strip_dongle_id
    removed_dongle_route_list = []
    for idx, row in df.iterrows():
        stripped_value = row['fullname'].replace(f'{DONGLE_ID}', 'INSERT-DONGLE-ID-HERE')
        removed_dongle_route_list.append(stripped_value)
    df['fullname'] = removed_dongle_route_list
    
    # df = df.sort_values('end_time_utc_millis', ascending=False)
    df = df.sort_values('length', ascending=False)
    route_names = df['fullname'].tolist()
    route_df = pd.DataFrame()
    route_df['route_name'] = route_names

    # Save route data to csv
    route_df.to_csv(f'{route_data_path}/route_names.csv', index=False)
    df.to_csv(f'{route_data_path}/trip_driving_data.csv', index=False)
    print(df.head(5))

query_route_data(BASE_URL=BASE_URL)
                                       fullname   length  create_time  \
15   INSERT-DONGLE-ID-HERE|000000a3--d4138141b2  170.033   1721761545   
398  INSERT-DONGLE-ID-HERE|2024-04-07--06-08-29  156.534   1712870311   
16   INSERT-DONGLE-ID-HERE|000000a2--f68bb5fd8b  141.713   1721760862   
2    INSERT-DONGLE-ID-HERE|000000b0--db57c5d3b3  140.854   1721967490   
318  INSERT-DONGLE-ID-HERE|2024-05-07--16-10-54  139.620   1715288166   

     end_time_utc_millis    end_time  init_logmonotime  maxqcamera  maxqlog  \
15         1721760797000  2024-07-23    49470961855087         111      111   
398        1712514891000  2024-04-07   599843939121748         146      146   
16         1721747729000  2024-07-23    37275863139030          96       96   
2          1721962915000  2024-07-26   253924928464360          72       72   
318        1715129896000  2024-05-08   111658668871310         227      227   

              platform  procqcamera  procqlog  \
15   TOYOTA_CAMRY_TSS2          111        -1   
398  TOYOTA CAMRY 2021          146       146   
16   TOYOTA_CAMRY_TSS2           96        -1   
2    TOYOTA_CAMRY_TSS2           72        -1   
318  TOYOTA CAMRY 2021          227       227   

                                     segment_end_times  \
15   [1721754172000, 1721754232000, 1721754292000, ...   
398  [1712506162000, 1712506222000, 1712506282000, ...   
16   [1721741977000, 1721742037000, 1721742097000, ...   
2    [1721958626000, 1721958686000, 1721958746000, ...   
318  [1715116312000, 1715116372000, 1715116432000, ...   

                                       segment_numbers  \
15   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...   
398  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...   
16   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...   
2    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...   
318  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...   

                                   segment_start_times  start_time_utc_millis  \
15   [1721754112000, 1721754172000, 1721754232000, ...          1721754112000   
398  [1712506102000, 1712506162000, 1712506222000, ...          1712506102000   
16   [1721741917000, 1721741977000, 1721742037000, ...          1721741917000   
2    [1721958566000, 1721958626000, 1721958686000, ...          1721958566000   
318  [1715116252000, 1715116312000, 1715116372000, ...          1715116252000   

           version  time_diff_millis  time_diff_seconds  time_diff_minutes  \
15   0.9.7-release           6685000             6685.0         111.416667   
398  0.9.6-release           8789000             8789.0         146.483333   
16   0.9.7-release           5812000             5812.0          96.866667   
2    0.9.7-release           4349000             4349.0          72.483333   
318  0.9.6-release          13644000            13644.0         227.400000   

     time_diff_hours  
15          1.856944  
398         2.441389  
16          1.614444  
2           1.208056  
318         3.790000

API Request #2 - Returns URLs To Download Video Files

Using the route name, we can submit our second API request to an endpoint storing the URLs of our downloadable video files (.ts file type). Before downloading our files, we store the URLs from the API response in a text file, so we can access the URL data locally.

def query_to_extract_urls(BASE_URL: str, route_name: str):
    df = pd.read_csv(f'{route_data_path}/route_names.csv')

    # Insert dongle ID into route name
    route_name_dongle_list = []
    for idx, row in df.iterrows():
        converted_route_name = row['route_name'].replace(
            'INSERT-DONGLE-ID-HERE', f'{DONGLE_ID}')
        route_name_dongle_list.append(converted_route_name)
    df['route_name'] = route_name_dongle_list

    download_recent_trip_vids = df.loc[df['route_name'] == route_name]
    download_recent_trip_vids = download_recent_trip_vids['route_name'].tolist()

    for route in tqdm(download_recent_trip_vids):
        with get(
            f'{BASE_URL}/v1/route/{route}/files', 
            headers=headers, verify=False, 
            stream=True, 
            timeout=10) as response:
            content = response.json()['qcameras']
            with open(
                f'{vid_urls_path}' + f'/{route.replace(f"{DONGLE_ID}|", "")}.txt',
                mode="wb") as file:
                for url in content:
                    file.write(
                        url.replace(
                            f"{DONGLE_ID}", 
                            "INSERT-DONGLE-ID-HERE").encode('utf-8') + ' \n'.encode('utf-8'))
    urls_list = []
    with open(
        f'{vid_urls_path}' + f'/{route.replace(f"{DONGLE_ID}|", "")}.txt',
        mode="r") as file:
        url_list = file.readlines()
        for url in url_list:
            urls_list.append(url)
    print("Total number of URLs to download:", len(urls_list))
    print("\n Preview 5 URLs:", *url_list[:5], sep='\n')

query_to_extract_urls(BASE_URL=BASE_URL, route_name=f'{DONGLE_ID}|2024-04-07--06-08-29')
Total number of URLs to download: 147

 Preview 5 URLs:
https://commadata2.blob.core.windows.net/qlog/INSERT-DONGLE-ID-HERE/2024-04-07--06-08-29/0/qcamera.ts?se=2024-07-26T23%3A26%3A31Z&sp=r&sv=2024-05-04&sr=b&rscd=attachment%3B%20filename%3DINSERT-DONGLE-ID-HERE_2024-04-07--06-08-29--0--qcamera.ts&sig=IeU/1rDLYDyNdO8IfWGuY7lkyqklbWEzXhk4oTXMpNA%3D 

https://commadata2.blob.core.windows.net/qlog/INSERT-DONGLE-ID-HERE/2024-04-07--06-08-29/1/qcamera.ts?se=2024-07-26T23%3A26%3A31Z&sp=r&sv=2024-05-04&sr=b&rscd=attachment%3B%20filename%3DINSERT-DONGLE-ID-HERE_2024-04-07--06-08-29--1--qcamera.ts&sig=Utd4haRw/MTRCj9IQ5FuGlG%2BJEw9GteLC1C4hvc6QXw%3D 

https://commadata2.blob.core.windows.net/qlog/INSERT-DONGLE-ID-HERE/2024-04-07--06-08-29/2/qcamera.ts?se=2024-07-26T23%3A26%3A31Z&sp=r&sv=2024-05-04&sr=b&rscd=attachment%3B%20filename%3DINSERT-DONGLE-ID-HERE_2024-04-07--06-08-29--2--qcamera.ts&sig=84r3II6Jzj2o2XoZarOZWtdAnBASVU3/0RG39MGIvFg%3D 

https://commadata2.blob.core.windows.net/qlog/INSERT-DONGLE-ID-HERE/2024-04-07--06-08-29/3/qcamera.ts?se=2024-07-26T23%3A26%3A31Z&sp=r&sv=2024-05-04&sr=b&rscd=attachment%3B%20filename%3DINSERT-DONGLE-ID-HERE_2024-04-07--06-08-29--3--qcamera.ts&sig=qrJkNwjhOzm4oHXnrxINYvF/yp6tT9SsS%2BFlFUEz934%3D 

https://commadata2.blob.core.windows.net/qlog/INSERT-DONGLE-ID-HERE/2024-04-07--06-08-29/4/qcamera.ts?se=2024-07-26T23%3A26%3A31Z&sp=r&sv=2024-05-04&sr=b&rscd=attachment%3B%20filename%3DINSERT-DONGLE-ID-HERE_2024-04-07--06-08-29--4--qcamera.ts&sig=qhgD2bNQInMmdex8Y82SnbhVq5qGgJX9qMttW0cIZbU%3D

Downloading Our Driving Video .ts Files

With our URLs stored locally in a text file, we can iterate over and request each URL to download and save our video files locally.
Note: You cannot run this function since I did not provide my API token or dongle id

def download_vid_files_from_url():
    for filename in tqdm(listdir(vid_urls_path)):
        print("Video URLs file:", vid_urls_path +  f'/{filename}')
        count = 0
        f = os.path.join(vid_urls_path, filename)
        file = open(f, 'rb')
        print("Beginning video downloads...")
        for url in tqdm(file):
            decode_url = url.decode('utf-8')
            url_insert_dongle_id = decode_url.replace(
                "INSERT-DONGLE-ID-HERE", f"{DONGLE_ID}")
            create_route_vid_path = filename.replace('.txt', '').replace(f'{DONGLE_ID}|', '')
            urllib.request.urlretrieve(
                url_insert_dongle_id, 
                vid_save_files_path +
                f'/{create_route_vid_path}' + 
                f'/x{str(count).rjust(3, "0")}_' + 
                f'{filename.replace(".txt", "").replace(f"{DONGLE_ID}|", "")}.ts')
            count += 1
        sleep(17)
        print("Video files successfully downloaded!")
        print("Total files downloaded:", count)
        
download_vid_files_from_url()
Video URLs file: data/vid-urls/2024-04-07--06-08-29.txt
Beginning video downloads...
Video files successfully downloaded!
Total files downloaded: 147

Converting File Type to MP4

After looping over the URLs to download our driving videos, we convert our video file type from .ts to .mp4 since it’s one of the most common file types for videos. We store the converted videos in a separate directory, so that we can loop over the 147 files without the original files making trouble.

def convert_ts_to_mp4(vid_clip_directory: str):
    route_directory = vid_save_files_path + vid_clip_directory
    if path.exists(mp4_directory + vid_clip_directory) == False: 
        mkdir(mp4_directory + vid_clip_directory)
    else: pass
    
    files_list = []
    for file in listdir(route_directory): files_list.append(file)
    files_list.sort()
    for filename in files_list:
        infile = route_directory + f'/{filename}'
        outfile = mp4_directory + f'/{vid_clip_directory}' + f'/{filename.replace(".ts", "")}.mp4'
        subprocess.run([
            'ffmpeg',
            '-i',
            infile,
            outfile,
        ])
# convert_ts_to_mp4(vid_clip_directory='/2024-04-07--06-08-29')

Concatenate The Video Clips

To facilitate the distribution of video data, Comma API splits our video data into short clips to reduce the memory size. Our objective is to capture images from our entire trip; therefore, we need to concatenate the 147 video files. Ideally, we’d prefer to create one MP4 from the concatenation. Due to storage size, we split the final trip into 4 parts. If we don’t split the video data in this manner, the file size would be too large and we wouldn’t be able to push the video to GitHub.

def concat_vid_clips(vid_clip_directory: str):
    vid_clips_list = []
    route_mp4_path = mp4_directory + vid_clip_directory

    files_list = []
    for file in listdir(route_mp4_path): files_list.append(file)
    files_list.sort()

    def multi_part_full_vid(video_title: str, start_range: int, end_range: int):
        for filename in files_list[start_range:end_range]:
            f = os.path.join(route_mp4_path, filename)
            vid_clip = VideoFileClip(f)
            vid_clips_list.append(vid_clip)
        final_clip = concatenate_videoclips(clips=vid_clips_list, method='chain')
        final_clip.write_videofile(f'{full_vid_path}' + f'/{video_title}.mp4')
        vid_clips_list.clear()

    multi_part_full_vid(video_title="trip_part_1", start_range=0, end_range=40)
    multi_part_full_vid(video_title="trip_part_2", start_range=41, end_range=80)
    multi_part_full_vid(video_title="trip_part_3", start_range=81, end_range=120)
    multi_part_full_vid(video_title="trip_part_4", start_range=121, end_range=147)

# concat_vid_clips(vid_clip_directory='/2024-04-07--06-08-29')

Save Images From The Video

Finally, we play the videos and save an Image every 2500 frames.

def save_frame_range(
    video_path: str, 
    start_frame: int, 
    stop_frame: int, 
    step_frame: int,
    dir_path: str, 
    basename: str, 
    ext='png'):

    cap = cv2.VideoCapture(video_path)

    if not cap.isOpened(): return

    makedirs(dir_path, exist_ok=True)
    base_path = os.path.join(dir_path, basename)

    digit = len(str(int(cap.get(cv2.CAP_PROP_FRAME_COUNT))))

    for n in range(start_frame, stop_frame, step_frame):
        cap.set(cv2.CAP_PROP_POS_FRAMES, n)
        ret, frame = cap.read()
        if ret: cv2.imwrite(f'{base_path}_{str(n).zfill(digit)}.{ext}', frame)
        else: return

save_frame_range(full_vid_path + '/trip_part_1.mp4', 0, 200000, 
                 2500, images_path, 'part1_video_img_frame')

save_frame_range(full_vid_path + '/trip_part_2.mp4', 0, 200000, 
                2500, images_path, 'part2_video_img_frame')

save_frame_range(full_vid_path + '/trip_part_3.mp4', 0, 200000, 
                 2500, images_path, 'part3_video_img_frame')

save_frame_range(full_vid_path + '/trip_part_4.mp4', 0, 200000, 
                 2500, images_path, 'part4_video_img_frame')

# image_dir = listdir(images_path)
# for image in image_dir: print(image)

College Station Trip: Part 1/4

Notable Images From Part 1/4

College Station

Water Tower

Country Road

Traffic Light

Plotting Images

cstat = f'{images_path}' + '/part1_video_img_frame_002500.png'
water_tower = f'{images_path}' + '/part4_video_img_frame_25000.png'
country_road = f'{images_path}' + '/part3_video_img_frame_10000.png'
traffic_light = f'{images_path}' + '/part1_video_img_frame_010000.png'

cstat_image = plt.imread(cstat, format='png')
water_tower_image = plt.imread(water_tower, format='png')
country_road_image = plt.imread(country_road, format='png')
traffic_light_image = plt.imread(traffic_light, format='png')

ax0 = isns.imgplot(cstat_image, cmap='seismic', gray=True)
ax1 = isns.imgplot(water_tower_image, cmap='seismic', gray=True)
ax2 = isns.imgplot(country_road_image, cmap='seismic', gray=True)
ax3 = isns.imgplot(traffic_light_image, cmap='seismic', gray=True)
plt.show()

Plotting Images - Histograms

plt.subplot(2,2,1)
plt.hist(cstat_image.ravel())
plt.subplot(2,2,2)
plt.hist(water_tower_image.ravel())
plt.subplot(2,2,3)
plt.hist(country_road_image.ravel())
plt.subplot(2,2,4)
plt.hist(traffic_light_image.ravel())
plt.show()

Plotting Images - Boxplots

plt.subplot(2,2,1)
plt.boxplot(cstat_image.ravel())
plt.subplot(2,2,2)
plt.boxplot(water_tower_image.ravel())
plt.subplot(2,2,3)
plt.boxplot(country_road_image.ravel())
plt.subplot(2,2,4)
plt.boxplot(traffic_light_image.ravel())
plt.show()