Expand Your Front End Skills With Machine Learning

6 min readMar 5, 2021

As a software engineer especially in front end fields, we usually becomes an enabler for product and business to create things. As times goes on, front end not only do UI Layout and API integration, now we can do machine learning right on our client side.

Here I am going to share my experiment on how to create computer vision model to determine your hand sign with Alphabet ASL (American Sign Language).
What i did here can also be done with any front end fields out there (iOS, Android, Web Platform), but i will use web platform as example because currently I am active in javascript.

Mandatory prerequisite to follow:

Basic understanding of project management on Google Cloud Platform.
Basic understanding of this Google Cloud Platform products:
- Cloud Storage
- AutoML Vision
- Cloud Shell
Strong understanding of your front end fields.
Strong understanding of ML library on your front end fields:
- TensorFlow JS (for web platform)
- TensorFlow Lite (for android)
- Core ML (for iOS)
Download the dataset here https://www.kaggle.com/grassknoted/asl-alphabet

In order to make it easy for you to follow this step by step, i’m going to use Google Cloud Shell so we will have the same environment no matter what your device is.

Upload our dataset into cloud shell

we need to upload our dataset and resources to cloud shell, so we can play on that environment. Just click the upload button and upload your file as a zip file.

and unzip your file with basic linux command:

unzip your_zip_name

and you will see this folders:

Enable AutoML API

We need to enable AutoML API, so we can use this products. Go to AI section and choose AI Platform > Dashboard, and click the button.

choose what kind of output that we want to use, for this case we choose the Image Classification (Single-label)

leave everything by default, and click create. And this process will auto create bucket on Cloud Storage for further use.

Upload our dataset to cloud storage

Use gsutil to upload the test and train folders to cloud storage. Sample command will look like this, and adjust with your environment.
This will takes time because we are uploading 1000+ sample data :D. Go take a coffee break.

gsutil -m cp -r ./asl_alphabet_train/asl_alphabet_train gs://cloud-ai-platform-xxxxxxxxxxx/train

Labelling our dataset

this will be the exhausting part, we need to give a label to all of our data set.

create a csv file, and do the Labelling with this kind of format:

gs://path_to_your_file/A/A1.jpg,A
.
.
gs://path_to_your_file/A/A3000.jpg,Ags://path_to_your_file/A/B1.jpg,B
.
.
gs://path_to_your_file/A/B3000.jpg,B

and you need to create a csv files for all the alphabet (A — Z) and the edge case. Use any of your favourite programming language to make it automated 😎

Upload all of those csv files to Cloud Storage with the gsutil. Back to AI Platform section, we need to fill all of our csv file location from Cloud Storage to AI Platform.

Click continue and you will be redirected to preview process, if all of our previous steps was right, you will see something like this. All of the image with label. Next we click Train New Model.

Train New Model

After you click Train New Model, the configuration pop up will show. For the training method, choose AutoML Edge. AutoML Edge means, that our output will be compatible with any front end fields machine learning library.

For the optimizer options, just choose whatever you want or fit your use case, the explanation and the tradeoff are written there.

For the training budget hours, just fill it 15 hours. I will explain later on the pricing section below

Start train your model, and go take another coffee break.

Train Result

After long wait of training time, you will see the training result and all the summary detail. If everything goes well, you will see high score accuracy.

Export our model

You will be given some options which platform you will be targeted. like i mentioned before, we can choose to any of front end fields out there. But for this part i’m going to choose tensorflow.js

Implement on the web browser

Time to code on the client side, i’m going to use nextjs + vercel to make it everything simple for code and deployment. Just create a new nextjs project, and install additional dependencies to implement our models.

"@tensorflow/tfjs": "^2.8.6",    
"@tensorflow/tfjs-automl": "^1.0.0",

And the output will be look like this.

Pricing

You need to be careful for this part, because to use Google Cloud Platform you need to bind your payment card into their billing. What i did here did not consume any money, because i used all the process within free tier.

Let me breakdown one by one:

Cloud AutoML

Cloud AutoML have 15 node hours for edge training. The free node hour is issued at the time of the first batch prediction is initiated. You have up to one year to use them. As long as you stay within this range, you are safe.

Cloud Storage

Cloud Storage offers you this free quota:

5 GB-months of Standard Storage
5,000 Class A Operations per month
50,000 Class B Operations per month
1 GB network egress from North America to all region destinations (excluding China and Australia) per month

As long as you stay within this range, you are safe.

Cloud Shell

Cloud Shell is available at no additional cost for Google Cloud customers.

Full code implementation:

https://github.com/TitusEfferian/asl_alphabet

References: