Getting Hands on with Google Cloud Platform

For a while now I’ve been looking for an excuse to actually use Google Cloud Platform’s various services. The options are endless and I don’t pretend to know what each product does. But what I did want to do was understand better how it was connected together, how to set it up, and actually solve a problem with it.

Question: How Busy is my Street?

Sitting in my living room, I wondered how busy my street was and if there were odd times, patterns or insights I could find? To do this I thought some form of car/person counting out my front window could be pretty interesting. Also I don’t want to be looking like this guy to figure this out.

Breaking it Down

To do this I figured I’d need a couple things:

1. An edge device & camera setup to be able to observe the street
2. Object detection model for detecting cars, people, bikes etc.
3. Object counting with the object detection output
4. Somewhere to push the data off the camera device
5. Some way to access, view and use the data

TLDR: To do this, the final system I built was setup like this and worked quite well, counted cars, recorded to a database and updated a dashboard all in real time.

1. Edge Device & Camera: Raspberry Pi + PiCamera

Luckily I have a Raspberry Pi 4 around which met the requirements for this. I set it up on my local WiFi network and configured VS Code to have live SSH editing directly on the Pi. This made it pretty easy to prototype code, test functionality and monitor the system.

One challenge I faced was how to visualize the camera feed and be able to try and validate the object tracking by counting cars from the camera feed while it was running. To do this I setup VNC Viewer to be able to access the Pi’s GUI remotely and by displaying the camera feed directly onto the output of the Pi I could troubleshoot and see what was happening.

2. Object Detection & Counting: Google Coral TPU & TensorFlow Lite

Luckily, I had recently got the Google Coral Tensor Processing Unity (TPU) setup to run with my Raspberry Pi.

For those not familiar, the Google Coral device is a line Google’s proprietary TPU application specific integrated circuit (ASIC) specifically for neural network machine learning. This is the same chip available through Google Cloud for accelerated machine learning training and inference, but is available to third parties. There are numerous versions for prototyping or for including in production products. The one I have is the USB accelerator below.

Google has designed the Coral line up to work with TensorFlow Lite their mobile & IoT flavor of the main TensorFlow library. THe main challenges Google’s Coral lineup and TensorFlow Lite solve are:

• Latency: no round trip to servers for inference/training
• Privacy: no personal data leaves the devices
• Connectivity: no internet connectivity required to run the systems
• Size: both model and binary size reductions enable edge inference
• Power Consumption: reducing network connections and optimized inference means low power requirements

Google provides a number of existing pre-trained models for various tasks such as object detection (very useful!), semantic segmentation, pose estimation and speech recognition (Google Coral Model Zoo).

Here is the Pi with camera and Coral attached, would have liked to have a 3D printed case for it it wouldn’t have kept with the quick & dirty approach!

I chose to use the Single Shot Detector (SSD) MobileNet V2 model which was trained on the Common Objects in Context (COCO) dataset with 90 objects present in the training set. On the Coral this is pre-defined for an input image size of 300x300 pixels and a latency of 7.4 ms. So we should be able to run inference at ~100Hz, more than adequate for counting cars and bikes!

I ran the model on the Raspberry Pi without the USB accelerator to compare how much of a speedup it gave and got around 10x speed up per inference, not too bad! Couldn’t figure out why I was getting ~18ms inference vs. the claimed 7.4 but presume it’s due to some of the communication overhead and pre/post processing of the results happening on the RPi.

When running the object detection model I needed to set the detection threshold for the inference, i.e. how confident do I want the model to be before it says “hey! we’ve got something here!”

The trade offs I considered were having to deal with more triggers and potential false positives (more on this later) or the alternative is missing objects passing by. I erred on the cautious side and left it low at a 70% threshold and would rather make sure I see SOMETHING then reduce and remove false positives later.

Now how well did it run? The MobileNet V2 model has a plethora of various reports benchmarking it [1][2]. For my purposes I cared about how often it would miss objects, or falsely classify objects. The counting aspect is covered next, but the detection process had some interesting results!

First, the model was consistently able to detect cars in various conditions. Similar with trucks as their own class.

Now for the more interesting anomalies. One that occured ~20 times over 5-6 days was a bus near windows on the house across the street was classified as train.

Train detection:

Also, certain scenarios caused a ‘person’ to be detected where on closer inspection never actually was there. Like this one where the front of a truck was detected.

A couple ways that I managed this was using the fast processing time given by the Coral accelerator and implemented a minimum number of consecutive frames required before an object was considered “detected”. This also came into play with setting the confidence interval as mentioned above and a value of 0.7 gave reasonably reliable results without missing too many objects.

It even worked well when there was rain all over the window and classified the bus correctly!

3. Object Counting: K.I.S.S.

I contemplated implementing a form of in-frame object tracking system to make sure cars in the cameras field of view weren’t detected and counted twice, but then thought that wasn’t keeping in line with the “quick & dirty” nature of this project and raised the “keep it simple stupid” approach!

To do this I watched most cars passing by at approximately the speed limit took ~1 second to pass the window. I also knew that people walked down the sidewalk and across the cameras field of view in ~10 seconds. With this I thought it would be pretty simple to implement some logic that kept track of the last time each object was detected.

So if a car was detected and another was detected less than 1.5s later than it was quite likely it was the same car and therefore the second detection shouldn’t be counted. This approach has the fault that it can miss multiple cars in the same frame, and if a car drives by exceptionally slow it will count it twice.

I decided to test it out, and watched the camera feed and counted cars myself, then compared to the running system and it was getting ~90% of the cars passing by. This was acceptable to me as I wasn’t terribly concerned about specific numbers but more trends over time but I will definitely come back to improve on this!

4. Data to the Cloud: Google Cloud SQL

Now that I have a stream of data coming in, I need somewhere to put it. Looking at the GCP options the two that were best suited were BigQuery amd CloudSQL.

BigQuery being best suited for managing and querying large datasets almost instantaneously and is more meant as an analytics engine. The GCP CloudSQL offers MySQL or PostgreSQL relational database options, the nice part being all the versioning, updating and management handled by Google.

I decided to setup a CloudSQL instance based on MySQL to push the object detection data to. Since this is unlikely to get into the “big data” scale I still wanted to get hands on with BigQuery.

I discovered that BigQuery can incorporate CloudSQL databases into it and still utilize most of the benefits of BigQuery (I think!) so I planned to use that for the analytics side.

5. Visualize & Use the Data: BigQuery + Data Studio

Now with the Pi running the object detection model in real time and pushing data to the CloudSQL database we get to the interesting stuff!

In BigQuery you can add in an external database and run queries from within the BQ interface which is pretty slick. I don’t believe this has the same performance as standard BQ analysis but it’s nice to keep it all in the same interface.

Next I wanted some way to monitor the data and so sticking within the GCP platform I tried out Data Studio. Data Studio is Googles fully connected, semi-automated dash boarding tool which seamlessly integrates with a lot of the GCP products like BigQuery. Below is the finished product built with Data Studio.

It was easy to get setup, dragging charts and putting functional diagrams into the board to maintain understanding of the data. I did have some issues with getting filters on the various object categories to remove some categories for the plots, e.g. having a single plot for just cars detected vs. people (hence the various “Available Fields” in the image below).

Finally, I have an end to end IoT pipeline that I can setup my camera at the front of my house and auto-magically be able to see on the dashboard how many cars, people, and even benches (?) passed by my house!

Results

Here are the final stats:

Now digging through the trends and data I found a couple interesting things:

1. At peak times, around 4-8PM, 250+ cars passed in front of my house per hour

2. It was a much less popular walking route with fewer than 3-8 people passing by per hour

3. This road is used for both city buses as well as school buses consistently, one passed every ~30 minutes

4. There was a surprising lack of cyclists detected, only a hand full over a couple days.

This may be due to there being a dedicated bike route two blocks over from the house, which this data may show that the route is being used well.

Now what did I learn from all of this?

There were a lot of things I picked up but here are a few of the main ones.

1. Google Cloud Platform can be Expensive!: Spent $200 in a week with poor configurations

When I first setup the CloudSQL instances I reviewed the instance setup and thought it looked reasonable and “minimal” since I would just be prototyping in it. This meant choosing the minimum storage of 100Gb SSD storage and a single 3.75 Gb virtual CPU to run the MySQL database.

After about a week of running the database I had been monitoring the Billing page of my GCP account. I noticed that the summary page showed “zero” cost but below there was a table that showed $200 of costs but my GCP free trial credits negated the costs. This got me concerned as the summary plot looked like this still, apparently no costs incurred:

After digging I found the option which was auto-applying apply my free-trial credits by default.

After un-selecting that, the cost summary looked a bit more like what I expected. This shows that for the primary days I was running the SQL server, storage, etc. it was charging my account ~$20 per day which was mildly concerning! I quickly dug into what was causing the steep costs for such a simple setup and found that the vCPU instance could be downgraded to a shared version with ~0.6Gb memory.

After cutting down the options I got it down to ~$0.70 per day which was a bit more acceptable to my wallet. During this process I also discovered the GCP pricing calculator which I will definitely be using in the future!

Google Cloud Platform Pricing Calculator

2. Coral USB Accelerator Works Best with Models less than ~8Mb in Size

When getting the Coral accelerator setup and running I was looking at alternative models and wondered why certain variants of models were included in the model zoo and others not? After digging a bit closer I found out that the accelerator has ~8Mb onboard SRAM which is what gets initialized when inferences start to run. I have not trained any custom models for the Coral but would be quite frustrating to get to the run stage on it with TensorFlow Lite to find out you need to redesign your models and retrain or pay the price in inference time.

From Googles Coral site for Object Detection models:

Now with that being said, they have Pose Estimation models available that are compiled for the Coral but are in the 20-30 Mb range. How they get around the onboard SRAM constraints is unclear but obviously it’s possible likely with some inference time costs.

3. Ensuring the system was operating all the time proved harder than expected

After getting the full pipeline connected, checking each component could talk to one another and finally getting those first data samples streaming through was satisfying! But I quickly found myself checking the dashboard from my phone and noticing it wasn’t receiving fresh data.

Checking on the Pi via SSH showed some obscure errors and failures were causing it to crash and not re-start the system.

I luckily got a good recommendation from a friend to dig into the Supervisor process control system. This monitors current running processes and you can configure it to manage/run tasks, scripts, etc. You can set it up whether you want it to auto-restart, restart after a delay, etc.

I’m sure anyone with more of a software engineering background is bumping their head in a “duh” motion but boy was it nice to not have to dig into the OS to do this!

The settings in Supervisor I ended up using were the following:

[program:[PROCESS_NAME]]
command=[PATH TO PYTHON VIRTUAL ENV TO RUN SCRIPTS] -u [PATH TO SCRIPT TO RUN]
directory=[LOCATION WHERE TO RUN FROM]
stderr_logfile=[PATH TO WHERE A LOG FILE WILL SAVE OUTPUTS TO FOR DEBUGGING]
user=[USER ON THE PI]
startsecs=[HOW LONG TO WAIT TO CHECK IF IT'S RUNNING]
startretries=[HOW MANY TIMES TO RETRY IF IT FAILS TO LAUNCH]
autostart=true [SET IT TO START AUTOMATICALLY UPON STARTUP]
autorestart=true [SET IT TO RESTART AUTOMATICALLY IF NOT RUNNING]
stopasgroup=true [STOP ALL PROCESSES RELATED TO THIS PROCESS]

Thanks for taking the time to read all of this and hope it was interesting and/or useful!

Stay tuned for more info and progress updates and follow me on LinkedIn or GitHub below.

LinkedIn: Ty Andrews GitHub: tieandrews