Complete-Data-Science-Toolkits
The overall objective of this toolkit is to provide and offer a free collection of data analysis and machine learning that is specifically suited for doing data science. Its purpose is to get you started in a matter of minutes. You can run this collections either in Jupyter notebook or python alone.
Features
Machine Learning
- Cross-Validation
- Evaluating Classification Metrics
- Evaluating Clustering Metrics
- Evaluating Regression Metrics
- Grid Search
- Preprocessing Encoding Categorical Features
- Preprocessing Binarization
- Preprocessing Imputing Missing Values
- Preprocessing Normalization
- Preprocessing StandardScaler
- Randomized Parameter Optimization
Numpy
- Adding, Removing, and Splitting Arrays
- Sorting arrays
- Matrix object
- Statistics Vector Math
- Structured Arrays
- Import, Export, Slicing, Indexing
- Data to from string
Pandas
- Complete pandas
- Groupby in Pandas
- Mapping
- Filtering
- Applying
Visualization
- BarPlots
- Customization Matplotlib
- Working with Image
- Working with text
Naming Conventions
- The naming convections I followed is:
- [yyyy-mm-dd-in-project-name-library].extention
- yyyy = stands for year
- mm = stands for month
- dd = stands for day
- in = my initial, for example: Saleban Olow = so
- library = numpy, pandas, sklearn, matplotlib
- project-name = each project name
- extention = .ipynb, .py, .html
- Example: 2017-25-11-so-cross-validation-sklearn.ipynb
Code Samples:
Cross Validation
from sklearn.model_selection import cross_val_score
model = SVC(kernel='linear', C=1)
# let's try it using cv
scores = cross_val_score(model, X, y, cv=5)
Grid Search
from sklearn.grid_search import GridSearchCV
params = {"n_neighbors": np.arange(1,5), "metric": ["euclidean", "cityblock"]}
grid = GridSearchCV(estimator=knn, param_grid=params)
grid.fit(X_train, y_train)
print(grid.best_score)
print(grid.best_estimator_.n_neighbors)
Preprocessing Imputing Missing Values
from sklearn.preprocessing import Imputer
impute = Imputer(missing_values = 0, strategy='mean', axis=0)
impute.fit_transform(X_train)
Randomized Parameter Optimization
from sklearn.grid_search import RandomizedSearchCV
params = {"n_neighbors" : range(1,5), "weights": ["uniform", "distance"]}
rsearch = RandomizedSearchCV(estimator=knn, param_distributions=params, cv=4, n_iter=8, random_state=5)
rsearch.fit(X_train, y_train)
print(rsearch.best_score_)
Model fitting supervised and unsupervised learning
#supervised learning
from sklearn import neighbors
knn = neighbors.KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
#unsupervised learning
from sklearn.decomposition import PCA
pca = PCA(n_components=0.95)
pca_model = pca.fit_transform(X_train)
Working with numpy arrays
import numpy as np
#appends values to end of arr
np.append(arr, values)
#inserts values into arr before index 2
np.insert(arr, 2, values)
Indexing and Slicing arrays
import numpy as np
#return the element at index 5
arr = np.array([[1,2,3,4,5,6,7]])
arr[5]
#returns the 2D array element on index
arr[2,5]
#assign array element on index 1 the value 4
arr[1] = 4
#assign array element on index [1][3] the value 10
arr[1,3] = 10
Creating DataFrame
import pandas as pd
#specify values for each rows and columns
df = pd.DataFrame(
[[4,7,10],
[5,8,11],
[6,9,12]],
index=[1,2,3],
columns=['a','b','c'])
groupby pandas
import pandas as pd
import pandas as pd
#return a groupby object, grouped by values in column named 'cities'
df.groupby(by="Cities")
handling missing values
import pandas as pd
#drop rows with any column having NA/null data.
df.dropna()
#replace all NA/null data with value
df.fillna(value)
Melt function
import pandas as pd
#most pandas methods return a DataFrame so that
#this improves readability of code
df = (pd.melt(df)
.rename(columns={'old_name':'new_name', 'old_name':'new_name'})
.query('new_name >= 200')
)
Save plot
mport matplotlib.pyplot as plt
#saves plot/figure to image
plt.savefig('pic_name.png')
Marker, lines
import matplotlib.pyplot as plt
#add * for every data point
plt.plot(x,y, marker='*')
#adds dot for every data point
plt.plot(x,y, marker='.')
Figures, Axis
import matplotlib.pyplot as plt
#a container that contains all plot elements
fig = plt.figures()
#Initializes subplot
fig.add_axes()
#A subplot is an axes on a grid system, rows-cols num
a = fig.add_subplot(222)
#adds subplot
fig, b = plt.subplots(nrows=3, ncols=2)
#creates subplot
ax = plt.subplots(2,2)
Working with text plot
import matplotlib.pyplot as plt
#places text at coordinates 1/1
plt.text(1,1, 'Example text', style='italic')
#annotate the point with coordinates xy with text
ax.annotate('some annotation', xy=(10,10))
#just put math formula
plt.title(r'$delta_i=20$',fontsize=10)
-
语义分析是关于分析观众的一般意见。 这可能是对一则新闻，电影或任何有关正在讨论的事项的推文的反应。 通常，此类反应来自社交媒体，并通过NLP分组到文件中进行分析。 我们将首先简单地定义正面和负面的单词。 然后采用一种方法来分析这些单词作为句子的一部分使用这些单词。 我们使用nltk中的sentiment_analyzer模块。 我们首先用一个单词进行分析，然后用配对单词进行分析，也称为双字组。 最
-
摊销分析涉及估计程序中的操作序列的运行时间，而不考虑输入值中的数据分布的跨度。 一个简单的例子是查找排序列表中的值比未排序列表中的值更快。 如果列表已经排序，则分布数据的方式无关紧要。 但是，当然列表的长度会产生影响，因为它决定了算法为获得最终结果而必须经过的步骤数。 因此，我们看到，如果获得排序列表的单个步骤的初始成本很高，则查找元素的后续步骤的成本变得相当低。 因此，摊销分析有助于我们找到一系
-
算法分析 算法的效率可以在实现之前和实现之后的两个不同阶段进行分析。 他们是以下 - A Priori Analysis - 这是对算法的理论分析。 通过假设所有其他因素（例如，处理器速度）是恒定的并且对实现没有影响来测量算法的效率。 A Posterior Analysis - 这是一种算法的实证分析。 所选算法使用编程语言实现。 然后在目标计算机上执行此操作。 在此分析中，收集了所需的运行时间
-
Analysis Options The "Analysis options" tab lets you configure how tokens are handled, and which types of tests are performed during the analysis. Token Handling These settings control how tokens are
-
Package Analysis 是一款识别各类恶意软件的分析工具，可用于捕捉和对抗对开源注册表的恶意攻击。 该工具由开源安全基金会 (OpenSSF)发布，其中包含一些组件来帮助分析开源的软件包，特别是寻找恶意软件。组件包含： 调度程序 —— 从 Package Feeds 为分析工作者创建作业。 分析（one-shot analyze and worker）—— 通过对每个包的静态和动态分析，
-
学习 koa 源码的整体架构，浅析koa洋葱模型原理和co原理 1. 前言 你好，我是若川，微信搜索「若川视野」关注我，专注前端技术分享。欢迎加我微信ruochuan12，加群交流学习。 这是学习源码整体架构系列第七篇。整体架构这词语好像有点大，姑且就算是源码整体结构吧，主要就是学习是代码整体结构，不深究其他不是主线的具体函数的实现。本篇文章学习的是实际仓库的代码。 本文仓库地址：git clon