Github Repo Stars Predictor

Overview

It's a github repo star predictor that tries to predict the stars ofany github repository having greater than 100 stars. It predicts based onthe owner/organization's status and activities (commits, forks, comments,branches, update rate, etc.) on the repository. Different types of models(Gradient boost, Deep neural network, etc) have been tested successfullyon the dataset we fetched from github apis.

Dataset

We used the github REST api and GraphQL api to collect data of repositorieshaving more than 100 stars. The data is available in the dataset directoryWe were able to collect the data faster using the Digital Ocean's multipleservers. So we thanks Digital Ocean for providingfree credits to students to use servers. For the details on dataset featuresrefer the summary section below.

Tools used

• Python 2.7
• Jupyter Notebook
• NumPy
• Sklearn
• Pandas
• Keras
• Cat Boost
• Matplot Lib
• seaborn

We also used Google Colab's GPU notebooks.So we thank to Google for starting thier colab project forproviding GPUs

Code details

Below is a brief description for the Code files/folder in repo.

Bash Script

• settingUpDOServer.sh
  filepath: scripts/bash/settingUpDOServer.sh
  This is used for configuring the digital ocean server

NodeJs scripts

• getting_repos_v2.js
  filepath: scripts/nodejs/getting_repos_v2.js
  This script fetches the basic info of repos having more than 100 stars using the Github REST API

• githubGraphQLApiCallsDO_V2.js
  filepath: scripts/nodejs/githubGraphQLApiCallsDO_V2.js
  This script fetches the complete info of the repositories that were fetched by the abovescript and uses the Github GraphQL API. It follows the approach of fetching the dataat the fixed rate defined in env file (eg. 730ms per request)

• githubGraphQLApiCallsDO_V3.js
  filepath: scripts/nodejs/githubGraphQLApiCallsDO_V3.js
  This script fetches the complete info of the repositories that were fetched by the abovescript and uses the Github GraphQL API. It follows the approach of requesting data fornext repo after receiving the response of the already sent request.

Python scripts

• json_to_csv.py
  filepath: scripts/python/json_to_csv.py
  This script converts the json data fetched from Github's GraphQL API in the above script to theequivalent csv file.

• merge.py
  filepath: scripts/python/merge.py
  This scripts merges all the data in multiple csv files to a single csv file

Jupyter Notebooks

• VisualizePreprocess.ipynb
  filepath: notebooks/VisualizePreprocess.ipynb
  We have done the feature engineering task in this notebook. It visualises the data and correspondinglycreates new features, modifies existing features and removes redundant features. For detailson features created, check the summary below

• training_models.ipynb
  filepath: notebooks/training_models.ipynb
  In this notebook, we trained different models with hyper parameter tuning on our dataset and compared their result in the end.For details on models trained, their prediction scores, etc. check the summary below.

Summary

In this project we have tried to predict the number of starsof a github repository that have more than 100 stars. For this we havetaken the github repository data from github REST api and GraphQL api.After generating the dataset we visualized and did some feature engineeringwith the dataset and after that , finally we come up to the stage where weapplied various models and predicted the model's scores on training andtest data.

Feature Engineering

There are total of 49 features before pre-processing. After pre-processing (adding new features, removal of redundant features andmodifying existing features) the count changes to 54. All the features are listed below.Some features after pre-processing may not be clear. Please refer to the VisualizePreprocess.ipynb notebook for details.

Original Features

Features after pre-processing

Models Trained

• Gradient Boost Regressor
• Cat Boost Regressor
• Random Forest Regressor
• Deep Neural Network

Evaluation Metrics

• R^2 score
  

Results