Introduction¶
About¶
Team : The Elite
- Vidhya Subramaniam
- Devleena Banerjee
- Priyank jha
- Karthik chiranjeevi
Objective¶
Building A Mood Classifier Based On Facial Expressions - With AI and Computer Vision
The objective of this study is to classify mood of the person from facial expressions Images are categorized in three classes namely sadness, fear and happiness based on the emotion shown in the facial expressions .
Dataset¶
The data consists of 48x48 pixel grayscale images of faces. The pixel values are stored in 2304 (48*48) columns. These column names start with pixel. Along with pixel values, there is emotion column that say about mood of the image.
The task is to categorize each face based on the emotion shown in the facial expression in to one of three categories.
Along with pixel values, aithon2020_level2_traning.csv dataset contains another column emotion that say about mood that is present in the image. This is the dataset you will use to train your model.
The final test set, which will be used to determine the winner of the competition, will be published later.
https://spotleai.sgp1.digitaloceanspaces.com/course/zip/aithon2020-level-2.zip
The training dataset is stored inside the data folder.
Rules of submission¶
You have to submit folder after zipped it.
The name of the folder is aithon2020_yourname_or_teamname
The folder must not contain any dataset.
If you have used any python libraries other than in default list, you have to mention in requirements.txt file.
There must be aithon_level2.py file inside the folder.
And, a function name ‘aithon_level2_api’ must be defined inside the file.
The function will be called to train and test your program.
The function name, signature and output type is fixed.
The first argument is file name that contains data for training.
The second argument is file name that contains data for test.
The function must return predicted value or emotion for each data in test dataset sequentially in a list. For example, ['sad', 'happy', 'fear', 'fear', ... , 'happy']
By default following, Python libraries are installed in test environment
- Numpy
- Pandas
- OpneCV for Python
- TensorFlows
- Keras
- PyTorch
- Scikit-learn
- Theano
- Matplotlib
- Seaborn
Importing packages and data¶
Import necessary packages¶
In [3]:
!pip install livelossplot
In [4]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import cv2 as cv
import os
from sklearn.model_selection import train_test_split
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import tensorflow as tf
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropout, BatchNormalization, Activation
from tensorflow.keras import Model,Sequential
from tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import plot_model
from livelossplot import PlotLossesKeras
from tensorflow.keras.models import load_model
from sklearn.metrics import confusion_matrix
import random
from google.colab import files
Downloading the data¶
Downloading dataset.zip
In [5]:
!wget https://spotleai.sgp1.digitaloceanspaces.com/course/zip/aithon2020-level-2.zip
Extracting the zip
In [6]:
!unzip aithon2020-level-2.zip
Import the training data¶
In [7]:
train = pd.read_csv('aithon2020-level-2/data/aithon2020_level2_traning.csv')
train.head()
Out[7]:
Generating images from given data¶
image_from_array() : Return images, when passed an array.¶
In [8]:
def image_from_array(arr,width,height,return_image=False):
"""
Input : Takes in an array, width and height
Output: Displays an image by reshaping image to
provided width and height.
# Press any key to close the window
# if return_image=True, then the image matrix is returned
# instead of displaying it.
"""
# Reshaping the given array
img = np.array(arr.reshape(width,height),dtype=np.uint8)
if return_image:
return img
else:
# displaying image ; press any button to close
cv.imshow('image',img)
cv.waitKey(0)
cv.destroyAllWindows()
generate_images(data,labels) : Creates a images folder and stores images¶
In [ ]:
def generate_images(data,labels):
"""
Input : Input data matrix of images and lables list
Output: Store all the images in the /images/ folder
along with labels.csv
"""
# Checking if given data matrix and labels match
assert len(data)==len(labels),"Input array size ≠labels size"
# Getting current working directory
path = os.getcwd()
label_list = []
try:
# Creating a new directory 'images'
os.mkdir(path+'/images')
except OSError as error:
# If directory already exists
print(error)
print("\nDelete the existing images folder & try again")
store_path = path+'/images/'
for i in range(len(data)):
img = image_from_array(data[i],48,48,return_image=True)
cv.imwrite(store_path+str(i)+'.png',img)
label_list.append([i,labels[i]])
label_df = pd.DataFrame(label_list,columns=['Image','Emotion'])
label_df.to_csv(store_path+'labels.csv',index=False)
In [ ]:
generate_images(train.iloc[:,1:].values,train['emotion'].values)
In [9]:
def generate_images_folderwise(x_train,y_train,x_test,y_test):
"""
Input : Input data matrix of images, lables list
Output: Store all the images in the
/images/<train or test>/<class_folder>
"""
# Checking if given data matrix and labels match
assert len(x_train)==len(y_train),"Input array size ≠labels size"
assert len(x_test)==len(y_test),"Input array size ≠labels size"
# Getting current working directory
path = os.getcwd()
temp = 0
try:
# Creating a new directory 'images'
os.mkdir(path+'/images')
os.mkdir(path+'/images/'+'train')
os.mkdir(path+'/images/'+'test')
except OSError as error:
# If directory already exists
print(error)
print("\nDelete the existing images folder & try again")
store_path = path+'/images/'+'train/'
class_list = np.unique(y_train)
for j in class_list:
temp = 0
os.mkdir(store_path+str(j))
for i in range(len(x_train)):
if y_train[i]==j:
temp += 1
img = image_from_array(x_train[i],48,48,return_image=True)
cv.imwrite(store_path+str(j+'/')+str(temp)+'.png',img)
store_path = path+'/images/'+'test/'
class_list = np.unique(y_test)
for j in class_list:
temp = 0
os.mkdir(store_path+str(j))
for i in range(len(x_test)):
if y_test[i]==j:
temp += 1
img = image_from_array(x_test[i],48,48,return_image=True)
cv.imwrite(store_path+str(j+'/')+str(temp)+'.png',img)
In [10]:
x = train.iloc[:,1:].values
y = train['emotion'].values
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=y,random_state=42)
x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.25,random_state=42)
generate_images_folderwise(x_train,y_train,x_val,y_val)
In [11]:
print("x_train : {0}, y_train : {1}".format(x_train.shape,y_train.shape))
print("x_val : {0}, y_val : {1}".format(x_val.shape,y_val.shape))
print("x_test : {0}, y_test : {1}".format(x_test.shape,y_test.shape))
Making a copy of x_test
In [12]:
x_test_copy = x_test
Exploratory data analysis¶
Checking for class imbalance¶
In [13]:
temp = train['emotion'].value_counts()
sns.barplot(x=temp.index,y=temp.values)
plt.title("Emotions v/s Number of points belonging to each class\n\n")
plt.xlabel("Emotion")
plt.ylabel("Number of images")
Out[13]:
In [14]:
class_label = train['emotion'].value_counts()
total_points = len(train)
print("Points with class label -> Happy are = ",class_label.values[0]/total_points*100,"%")
print("Points with class label -> Sad are = ",class_label.values[1]/total_points*100,"%")
print("Points with class label -> Fear are = ",class_label.values[2]/total_points*100,"%")
labels = ['Happy','Sad','Fear']
sizes = [44.45,30.23,25.30]
colors = ['yellowgreen', 'gold','blue']
plt.figure(figsize=(6,6))
plt.title("% of points belonging to each class\n\n")
plt.pie(sizes, labels=labels, colors=colors,autopct='%1.1f%%', shadow=True)
Out[14]:
Exploring various images from the dataset¶
The referecne for the code below is taken from :
https://www.kaggle.com/xhlulu/eda-simple-keras-cnn-k-mnist
The below code loads 25 random images from dataset, with their label.
In [15]:
plt.figure(figsize=(10,10))
for i in range(25):
temp = np.random.randint(0,10000)
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([])
plt.grid(True)
plt.imshow(image_from_array(train.iloc[temp,1:].values,48,48,return_image=True),cmap="gray")
plt.xlabel(train['emotion'].values[temp])
Preparing data for modeling¶
Data Augmentation¶
The reference for below code was taken from : https://www.kaggle.com/sanikamal/multi-class-image-classification-with-augmentation
In [16]:
path = os.getcwd()
TRAINING_DIR = path+"/images/train/"
training_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
VALIDATION_DIR = path+"/images/test/"
validation_datagen = ImageDataGenerator(rescale = 1./255)
train_generator = training_datagen.flow_from_directory(
TRAINING_DIR,
color_mode='grayscale',
target_size = (48,48),
batch_size = 64,
class_mode='categorical',
shuffle=True)
validation_generator = validation_datagen.flow_from_directory(
VALIDATION_DIR,
color_mode = 'grayscale',
target_size = (48,48),
batch_size = 64,
class_mode='categorical',
shuffle=False)
Modeling¶
A simple CNN model¶
Loading the initial parameters
In [17]:
num_classes = 3
epochs = 100
In [18]:
tf.keras.backend.clear_session()
In [19]:
# Creating the model
model = Sequential()
model.add(Conv2D(32, kernel_size=(5, 5), input_shape=(48,48,1),activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(32, kernel_size=(5, 5),activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Dropout(0.25))
model.add(Conv2D(64, kernel_size=(3, 3),activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(64, kernel_size=(3, 3),activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(3, 3),strides=2))
model.add(Dropout(0.25))
model.add(Conv2D(128, kernel_size=(3, 3),activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(128, kernel_size=(3, 3),activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(1, 1),strides=2))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.summary()
In [20]:
model.compile(
loss = 'categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
In [21]:
steps_per_epoch = train_generator.n//train_generator.batch_size
validation_steps = validation_generator.n//validation_generator.batch_size
checkpoint = ModelCheckpoint("model_weights.h5", monitor='val_accuracy',
save_weights_only=True, mode='max', verbose=1)
callbacks = [PlotLossesKeras(), checkpoint]
history = model.fit(
x=train_generator,
steps_per_epoch=steps_per_epoch,
epochs=epochs,
validation_data = validation_generator,
validation_steps = validation_steps,
callbacks=callbacks
)
Saving the model¶
In [22]:
model.save('model')
In [23]:
!zip -r model.zip model
In [24]:
files.download('model.zip')
Evaluating the model¶
In [25]:
x_test = x_test.astype('float32')/255.0
In [26]:
train_generator.class_indices
Out[26]:
In [27]:
def evaluate(x_test,return_misclassified=False,y_test=[]):
output = []
misclassified_indices = []
count = 0
temp = 0
for i in range(len(x_test)):
temp = np.argmax(model.predict(x_test[i].reshape(1,48,48,1)),axis=1)
if temp ==0:
output.append('Fear')
elif temp==1:
output.append('Happy')
elif temp==2:
output.append('Sad')
if return_misclassified:
if output[i]!=y_test[i]:
misclassified_indices.append(i)
if return_misclassified:
return output,misclassified_indices
else:
return output
In [28]:
output = evaluate(x_test)
In [29]:
print("Accuracy on unseen data : ",np.sum(output==y_test)/len(x_test))
Understanding the model in depth¶
Where the model is going wrong¶
Getting points from data on which model is not working
In [30]:
output, id = evaluate(x_test,return_misclassified=True,y_test=y_test)
In [31]:
#plt.subplot_tool() -> Use this if you are running in local jupyter-notebook and not colab
plt.figure(figsize=(10,12))
for i in range(25):
temp = random.sample(id,1)[0]
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([])
plt.grid(True)
plt.imshow(image_from_array(x_test_copy[temp],48,48,return_image=True),cmap="gray")
plt.xlabel("Actual :"+y_test[temp]+'\n'+"Predicted :"+output[temp])
Confusion matrix¶
In [32]:
cm = confusion_matrix(y_test,output)
In [33]:
cm = pd.DataFrame(cm, columns=['Fear','Happy','Sad'],index=['Fear','Happy','Sad'])
In [35]:
plt.xlabel("Predicted")
plt.ylabel("Actual")
sns.heatmap(cm,annot=True,fmt="d")
Out[35]:
