初めてのPygameプログラムの作り方

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import lightgbm as lgb
import gc
%matplotlib inline 

path = '/home/WLY/learn/Kaggle_example_learn/Instacart/Data/'
aisles = pd.read_csv(path + 'aisles.csv')
departments = pd.read_csv(path + 'departments.csv')
products = pd.read_csv(path + 'products.csv')
orders = pd.read_csv(path + 'orders.csv')
order_products__train = pd.read_csv(path + 'order_products__train.csv')
order_products__prior = pd.read_csv(path + 'order_products__prior.csv')

Create a new DataFrame to store user features
user_fea=pd.DataFrame()
Add a new user id column to get the value of the user id
user_fea['user_id']=orders['user_id'].unique()
Sort the values of the user ids
user_fea=user_fea[['user_id']].sort_values('user_id')

Group the orders dataset by user id and count the number of order_id's, i.e. get the total number of user purchases
user_fea['user_orderid_count']=orders.groupby('user_id')['order_id'].count().values

The days_since_prior_order user in the orders dataset is based on the time interval between this single purchase and the previous one to derive the following.
1. If the average number of days between user purchases is small, it means that the user is a frequent shopper and is likely to keep consuming certain products
2. If the maximum number of days since the last purchase is small, the user loves to shop very much
3. If the variance of the user's time since the last purchase is small, it means that the user has a strong periodicity of shopping
4. If the plural of the user's time since last purchase is small, it proves that the user loves shopping
user_fea['user_days_since_prior_order_mean']=orders.groupby('user_id')['days_since_prior_order'].mean().values

user_fea['user_days_since_prior_order_max'] = orders.groupby('user_id')['days_since_prior_order'].max().values

user_fea['user_days_since_prior_order_std'] = orders.groupby('user_id')['days_since_prior_order'].std().values

user_fea['user_days_since_prior_order_mode'] = orders.groupby('user_id')['days_since_prior_order'].apply(lambda x: x.mode()[0]).values
user_fea.head()

The following scenario is derived from the order_dow and order_hour_of_day user's days of the week and points in time purchased in the orders dataset.
According to the plurality of the number of days of the week and the point in time of the user's purchase, we may know some of the user's habits, etc.
user_fea['user_order_dow_mode'] = orders.groupby('user_id')['order_dow'].apply(lambda x: x.mode()[0]).values
user_fea['user_order_hour_of_day_mode'] = orders.groupby('user_id')['order_hour_of_day'].apply(lambda x: x.mode()[0]).values

Add a second-order user time preference feature to indicate what time of the week the user prefers to start shopping? Still have some questions about this feature. 
orders['dow_hour'] = orders['order_dow'].values * 25 + orders['order_hour_of_day'].values
user_fea['user_dow_hour_mode'] = orders.groupby('user_id')['dow_hour'].apply(lambda x: x.mode()[0]).values
user_fea.head()

Note that the product features are extracted from the priors dataset here, why not combine it with train?
The reason is that this is a much larger data set, mainly because there are no similar features in the test, and the test user does not appear in the train.
Therefore, it is problematic to extract features in this way, because the features in the training set come from priors and trains, but the test only comes from priors, which may be biased.
Consider.
By splicing order_product__prior_and_orders, the following scenario is obtained based on the spliced dataset.
The number of different products purchased by users, i.e. the types of products
The three products that users buy the most, and if users buy only two types of products, the last product is represented by -1
The number of all products purchased by the user
The average number of products purchased by users per order

order_product_prior_=order_products__prior.merge(orders,on='order_id',how='left')

Group the prior_ data set by user id and count the number of different products, i.e. the number of product types
user_fea['user_product_nunique']=order_product_prior_.groupby('user_id')['product_id'].nunique().sort_index().values

Define functions to calculate the plurality for up to three product cases
def mode_N(x,i):
    m = x.value_counts().index
    if len(m) > i: 
        return m[i] 
    return -1
user_fea['user_product_mode'] = order_product_prior_.groupby('user_id')['product_id'].apply(lambda x: x.mode()[0]).sort_index().values
user_fea['user_product_mode_1'] = order_product_prior_.groupby('user_id')['product_id'].apply(lambda x: mode_N(x,1)).sort_index(). values 
user_fea['user_product_mode_2'] = order_product_prior_.groupby('user_id')['product_id'].apply(lambda x: mode_N(x,2)).sort_index(). values

Counting the number of product ids according to the grouping of user ids according to the prior_dataset, i.e. getting the total number of all products purchased by users
user_fea['user_product_count']=order_product_prior_.groupby('user_id')['product_id'].count().sort_index().values

The total number of products purchased by the user / the total number of purchases made by the user, i.e. the average number of products purchased by the user per order
user_fea['user_product_orderid_ratio'] = user_fea['user_product_count'] / user_fea['user_orderid_count']

user_fea.head()

Group the dataset by user id and order id to get the maximum value of each order id added to the basket order
1. the maximum number of products in the user's order
2. the average amount of products purchased by the user per order
3. 1/4, 3/4 digits of the products purchased by the user

tmp=order_product_prior_.groupby(['user_id','order_id'])['add_to_cart_order'].max().reset_index()

Group the newly constructed table according to the user id and get the maximum value added to the order of the shopping cart to get the maximum number of items purchased by the user in all orders
user_fea['user_add_to_cart_order_max'] = tmp.groupby('user_id')['add_to_cart_order'].max().sort_index().values

Group the newly constructed table according to the user id and get the average of the order added to the cart to get the average number of items purchased by the user each time
user_fea['user_add_to_cart_order_mean'] = tmp.groupby('user_id')['add_to_cart_order'].mean().sort_index().values

Group the newly constructed table according to the user id and get the 1/4,3/4 digits added to the order of the shopping cart, so that we can get the 1/4 and 3/4 digits of the number of products purchased by the user
user_fea['user_add_to_cart_order_quantile_25'] = tmp.groupby('user_id')['add_to_cart_order'].quantile().sort_index().values
user_fea['user_add_to_cart_order_quantile_75'] = tmp.groupby('user_id')['add_to_cart_order'].quantile(0.75).sort_index().values

Free memory
del tmp
gc.collect()

Get the sum and mean of the repurchase cases based on user id grouping
user_fea['user_reordered_sum'] = order_product_prior_.groupby('user_id')['reordered'].sum().sort_index().values
user_fea['user_reordered_mean'] = order_product_prior_.groupby('user_id')['reordered'].mean().sort_index().values