Loading IMDB data into SQLite in 200 lines of Python

You will be surprised how incredibly useful and powerful SQLite is.

By Vamshi Jandhyala in data management python

October 6, 2019

My wife and I love watching movies(mainly Telugu and Hindi) during weekends. I realized that we were wasting a lot of time reading reviews on the internet trying to decide which movies to watch. There definitely had to be a better way!

I knew that IMDB ratings were pretty reliable so I wondered if I could lay my hands on the IMDB ratings data. After doing some research on the internet, I stumbled upon this site IMDB datasets.

The data seemed fairly normalized so I wrote a quick and dirty script in python which downloads the zipped data files, unzips it and loads them into an SQLite database.

I am fairly happy with the result, it takes roughly 45 mins end to end from the point of downloading the files to loading the data into the database without any manual intervention.

Here are some highly rated Indian movies in 2019.

Original TitleGenres
JokerCrime,Drama,Thriller
Ardaas KaraanDrama
Family of ThakurganjAction,Drama
The Sound StoryDocumentary,Drama
ShibuComedy
Oththa Seruppu Size 7Drama
AsuranAction,Drama
DigbhayamThriller
Anandi GopalBiography
MoothonCrime,Drama
GirgitComedy
AddaComedy
Avengers: EndgameAction,Adventure,Sci-Fi
Once Upon a Time… in HollywoodComedy,Drama
Toy Story 4Adventure,Animation,Comedy
Uri: The Surgical StrikeAction,Drama,War
Gully BoyDrama,Music
Super 30Biography,Drama
Article 15Crime,Drama
ChhichhoreComedy,Drama
The Tashkent FilesDrama,Mystery,Thriller
Super DeluxeAction,Crime,Drama
Nerkonda PaarvaiAction,Drama,Thriller
Kumbalangi NightsComedy,Drama,Romance
JerseyDrama,Sport
YatraBiography,Drama
Agent Sai Srinivasa AthreyaAction,Comedy,Crime
Sye Raa Narasimha ReddyAction,Drama,History
UndaAction,Comedy,Thriller
VirusDrama,Thriller
The IrishmanBiography,Crime,Drama
EvaruCrime,Drama,Thriller
Portrait de la jeune fille en feuDrama,History,Romance
Thanneer Mathan DhinangalComedy,Romance
UyareDrama
Section 375Crime,Drama
BrochevarevaruraComedy
Seetharama KalyanaAction,Drama
KavaludaariThriller
LucaDrama,Mystery,Romance
The GambinosCrime,Drama
Gang LeaderAction,Comedy
PengalilaDrama
RakshasuduAction,Crime,Thriller
JiiviDrama,Thriller
ThamaashaComedy,Drama
Chaal Jeevi LaiyeDrama
To LetFamily
Padmavyuhathile AbhimanyuDrama
IlayarajaDrama
Chal Mera PuttComedy,Drama
GanagandharvanComedy,Drama,Music
JallikattuCrime
MalleshamBiography
Ninnu ThalachiRomance
KontthoDrama
ChasaniDrama,Romance
Sikander 2Drama
Vinara sodara veera kumaraDrama,Romance
PenaltySport
NetworkDrama,Thriller
SiegeSci-Fi,Thriller
Kempe Gowda 2Action,Thriller
Arjun SuravaramCrime
Super DuperComedy,Thriller
The Great Indian EscapeAdventure,Biography,Drama
Has Jhan Pagli Fas JabeRomance
NoblemenDrama
Nanna PrakaraCrime,Mystery
AamisDrama
Hansa Ek SanyogDrama
Jacqueline I Am ComingDrama
PratibhaDrama
The Himalayan DiariesComedy,Romance

Here is the script: gist