update readme and include example scripts

README.md

@@ -0,0 +1,72 @@
+# AIDE: Autonomous AI for Data Science
+Welcome to the official repository for AIDE, an AI system that can automatically solve data science tasks at a human level, and with human input, it can perform even better. We believe giving developers and researchers direct access to AIDE locally, with local compute and choice to use their own LLM keys, is the most straightforward way to make it useful. That's why we'll open-source it, and the tentative timeline is it will arrive before the end of April. Currently, this repository serves as a gallery showcasing its solutions for 60+ Kaggle competitions we tested.
+
+## About AIDE
+AIDE is an AI-powered data science assistant that can autonomously understand task requirements, design, and implement solutions. By leveraging large language models and innovative agent architectures, such as the Solution Space Tree Search algorithm, AIDE has achieved human-level performance on a wide range of data science tasks, outperforming over 50% of human data scientists on Kaggle competitions.
+
+## Gallary
+| task_name                                     |   top%_mean | file_link                                                                                                     | competition_link                                                                                       |
+|:----------------------------------------------|------------:|:--------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------|
+| bike-sharing-demand                           |        0.24 | [bike-sharing-demand.py](examples/bike-sharing-demand.py)                                                     | [competition link](www.kaggle.com/competitions/bike-sharing-demand/overview)                           |
+| tabular-playground-series-jul-2021            |        0.16 | [tabular-playground-series-jul-2021.py](examples/tabular-playground-series-jul-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-jul-2021/overview)            |
+| tabular-playground-series-jan-2022            |        0.48 | [tabular-playground-series-jan-2022.py](examples/tabular-playground-series-jan-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-jan-2022/overview)            |
+| tabular-playground-series-feb-2022            |        0.25 | [tabular-playground-series-feb-2022.py](examples/tabular-playground-series-feb-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-feb-2022/overview)            |
+| tabular-playground-series-feb-2021            |        0.71 | [tabular-playground-series-feb-2021.py](examples/tabular-playground-series-feb-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-feb-2021/overview)            |
+| tabular-playground-series-aug-2022            |        0.59 | [tabular-playground-series-aug-2022.py](examples/tabular-playground-series-aug-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview)            |
+| playground-series-s3e24                       |        0.55 | [playground-series-s3e24.py](examples/playground-series-s3e24.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e24/overview)                       |
+| playground-series-s3e23                       |        0.2  | [playground-series-s3e23.py](examples/playground-series-s3e23.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e23/overview)                       |
+| playground-series-s3e22                       |        0.87 | [playground-series-s3e22.py](examples/playground-series-s3e22.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e22/overview)                       |
+| playground-series-s3e19                       |        0.18 | [playground-series-s3e19.py](examples/playground-series-s3e19.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e19/overview)                       |
+| playground-series-s3e18                       |        0.26 | [playground-series-s3e18.py](examples/playground-series-s3e18.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e18/overview)                       |
+| playground-series-s3e17                       |        0.46 | [playground-series-s3e17.py](examples/playground-series-s3e17.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e17/overview)                       |
+| playground-series-s3e16                       |        0.64 | [playground-series-s3e16.py](examples/playground-series-s3e16.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e16/overview)                       |
+| playground-series-s3e14                       |        0.71 | [playground-series-s3e14.py](examples/playground-series-s3e14.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e14/overview)                       |
+| optiver-trading-at-the-close                  |        0.99 | [optiver-trading-at-the-close.py](examples/optiver-trading-at-the-close.py)                                   | [competition link](www.kaggle.com/competitions/optiver-trading-at-the-close/overview)                  |
+| new-york-city-taxi-fare-prediction            |        1    | [new-york-city-taxi-fare-prediction.py](examples/new-york-city-taxi-fare-prediction.py)                       | [competition link](www.kaggle.com/competitions/new-york-city-taxi-fare-prediction/overview)            |
+| playground-series-s3e25                       |        0.79 | [playground-series-s3e25.py](examples/playground-series-s3e25.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e25/overview)                       |
+| tmdb-box-office-prediction                    |        0.68 | [tmdb-box-office-prediction.py](examples/tmdb-box-office-prediction.py)                                       | [competition link](www.kaggle.com/competitions/tmdb-box-office-prediction/overview)                    |
+| icr-identify-age-related-conditions           |        0.91 | [icr-identify-age-related-conditions.py](examples/icr-identify-age-related-conditions.py)                     | [competition link](www.kaggle.com/competitions/icr-identify-age-related-conditions/overview)           |
+| house-prices-advanced-regression-techniques   |        0.41 | [house-prices-advanced-regression-techniques.py](examples/house-prices-advanced-regression-techniques.py)     | [competition link](www.kaggle.com/competitions/house-prices-advanced-regression-techniques/overview)   |
+| godaddy-microbusiness-density-forecasting     |        0.65 | [godaddy-microbusiness-density-forecasting.py](examples/godaddy-microbusiness-density-forecasting.py)         | [competition link](www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/overview)     |
+| cat-in-the-dat                                |        0.69 | [cat-in-the-dat.py](examples/cat-in-the-dat.py)                                                               | [competition link](www.kaggle.com/competitions/cat-in-the-dat/overview)                                |
+| tabular-playground-series-apr-2021            |        0.77 | [tabular-playground-series-apr-2021.py](examples/tabular-playground-series-apr-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-apr-2021/overview)            |
+| tabular-playground-series-apr-2022            |        0.51 | [tabular-playground-series-apr-2022.py](examples/tabular-playground-series-apr-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-apr-2022/overview)            |
+| tabular-playground-series-aug-2021            |        0.16 | [tabular-playground-series-aug-2021.py](examples/tabular-playground-series-aug-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-aug-2021/overview)            |
+| ciphertext-challenge-iii                      |        0.91 | [ciphertext-challenge-iii.py](examples/ciphertext-challenge-iii.py)                                           | [competition link](www.kaggle.com/competitions/ciphertext-challenge-iii/overview)                      |
+| ciphertext-challenge-ii                       |      nan    | [ciphertext-challenge-ii.py](examples/ciphertext-challenge-ii.py)                                             | [competition link](www.kaggle.com/competitions/ciphertext-challenge-ii/overview)                       |
+| cat-in-the-dat-ii                             |        0.66 | [cat-in-the-dat-ii.py](examples/cat-in-the-dat-ii.py)                                                         | [competition link](www.kaggle.com/competitions/cat-in-the-dat-ii/overview)                             |
+| tabular-playground-series-jan-2021            |        0.57 | [tabular-playground-series-jan-2021.py](examples/tabular-playground-series-jan-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-jan-2021/overview)            |
+| career-con-2019                               |        0.99 | [career-con-2019.py](examples/career-con-2019.py)                                                             | [competition link](www.kaggle.com/competitions/career-con-2019/overview)                               |
+| tabular-playground-series-jun-2022            |        0.7  | [tabular-playground-series-jun-2022.py](examples/tabular-playground-series-jun-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-jun-2022/overview)            |
+| spaceship-titanic                             |        0.61 | [spaceship-titanic.py](examples/spaceship-titanic.py)                                                         | [competition link](www.kaggle.com/competitions/spaceship-titanic/overview)                             |
+| tabular-playground-series-mar-2021            |        0.13 | [tabular-playground-series-mar-2021.py](examples/tabular-playground-series-mar-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-mar-2021/overview)            |
+| tabular-playground-series-mar-2022            |        0.85 | [tabular-playground-series-mar-2022.py](examples/tabular-playground-series-mar-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-mar-2022/overview)            |
+| tabular-playground-series-may-2021            |      nan    | [tabular-playground-series-may-2021.py](examples/tabular-playground-series-may-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-may-2021/overview)            |
+| tabular-playground-series-may-2022            |        0.6  | [tabular-playground-series-may-2022.py](examples/tabular-playground-series-may-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-may-2022/overview)            |
+| tabular-playground-series-oct-2021            |        0.62 | [tabular-playground-series-oct-2021.py](examples/tabular-playground-series-oct-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-oct-2021/overview)            |
+| tabular-playground-series-oct-2022            |        0.54 | [tabular-playground-series-oct-2022.py](examples/tabular-playground-series-oct-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-oct-2022/overview)            |
+| tabular-playground-series-sep-2021            |        0.62 | [tabular-playground-series-sep-2021.py](examples/tabular-playground-series-sep-2021.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-sep-2021/overview)            |
+| tabular-playground-series-jul-2022            |        0.95 | [tabular-playground-series-jul-2022.py](examples/tabular-playground-series-jul-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-jul-2022/overview)            |
+| sentiment-analysis-on-movie-reviews           |        0.42 | [sentiment-analysis-on-movie-reviews.py](examples/sentiment-analysis-on-movie-reviews.py)                     | [competition link](www.kaggle.com/competitions/sentiment-analysis-on-movie-reviews/overview)           |
+| nlp-getting-started                           |        0.48 | [nlp-getting-started.py](examples/nlp-getting-started.py)                                                     | [competition link](www.kaggle.com/competitions/nlp-getting-started/overview)                           |
+| jigsaw-toxic-comment-classification-challenge |        0.78 | [jigsaw-toxic-comment-classification-challenge.py](examples/jigsaw-toxic-comment-classification-challenge.py) | [competition link](www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/overview) |
+| playground-series-s3e1                        |        0.6  | [playground-series-s3e1.py](examples/playground-series-s3e1.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s3e1/overview)                        |
+| playground-series-s3e13                       |        0.03 | [playground-series-s3e13.py](examples/playground-series-s3e13.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e13/overview)                       |
+| playground-series-s3e15                       |        0.56 | [playground-series-s3e15.py](examples/playground-series-s3e15.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e15/overview)                       |
+| home-data-for-ml-course                       |        0.09 | [home-data-for-ml-course.py](examples/home-data-for-ml-course.py)                                             | [competition link](www.kaggle.com/competitions/home-data-for-ml-course/overview)                       |
+| forest-cover-type-prediction                  |        0.47 | [forest-cover-type-prediction.py](examples/forest-cover-type-prediction.py)                                   | [competition link](www.kaggle.com/competitions/forest-cover-type-prediction/overview)                  |
+| facial-keypoints-detection                    |      nan    | [facial-keypoints-detection.py](examples/facial-keypoints-detection.py)                                       | [competition link](www.kaggle.com/competitions/facial-keypoints-detection/overview)                    |
+| playground-series-s3e20                       |      nan    | [playground-series-s3e20.py](examples/playground-series-s3e20.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e20/overview)                       |
+| dont-overfit-ii                               |        0.98 | [dont-overfit-ii.py](examples/dont-overfit-ii.py)                                                             | [competition link](www.kaggle.com/competitions/dont-overfit-ii/overview)                               |
+| scrabble-player-rating                        |      nan    | [scrabble-player-rating.py](examples/scrabble-player-rating.py)                                               | [competition link](www.kaggle.com/competitions/scrabble-player-rating/overview)                        |
+| digit-recognizer                              |        0.14 | [digit-recognizer.py](examples/digit-recognizer.py)                                                           | [competition link](www.kaggle.com/competitions/digit-recognizer/overview)                              |
+| tabular-playground-series-sep-2022            |        0.64 | [tabular-playground-series-sep-2022.py](examples/tabular-playground-series-sep-2022.py)                       | [competition link](www.kaggle.com/competitions/tabular-playground-series-sep-2022/overview)            |
+| playground-series-s3e26                       |        0.56 | [playground-series-s3e26.py](examples/playground-series-s3e26.py)                                             | [competition link](www.kaggle.com/competitions/playground-series-s3e26/overview)                       |
+| playground-series-s3e3                        |      nan    | [playground-series-s3e3.py](examples/playground-series-s3e3.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s3e3/overview)                        |
+| playground-series-s3e5                        |        0.61 | [playground-series-s3e5.py](examples/playground-series-s3e5.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s3e5/overview)                        |
+| playground-series-s3e7                        |        0.55 | [playground-series-s3e7.py](examples/playground-series-s3e7.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s3e7/overview)                        |
+| playground-series-s3e9                        |        0.86 | [playground-series-s3e9.py](examples/playground-series-s3e9.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s3e9/overview)                        |
+| playground-series-s4e1                        |        0.24 | [playground-series-s4e1.py](examples/playground-series-s4e1.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s4e1/overview)                        |
+| playground-series-s4e2                        |        0.64 | [playground-series-s4e2.py](examples/playground-series-s4e2.py)                                               | [competition link](www.kaggle.com/competitions/playground-series-s4e2/overview)                        |
+| competitive-data-science-predict-future-sales |        0.87 | [competitive-data-science-predict-future-sales.py](examples/competitive-data-science-predict-future-sales.py) | [competition link](www.kaggle.com/competitions/competitive-data-science-predict-future-sales/overview) |
+| santa-2019-revenge-of-the-accountants         |        0.96 | [santa-2019-revenge-of-the-accountants.py](examples/santa-2019-revenge-of-the-accountants.py)                 | [competition link](www.kaggle.com/competitions/santa-2019-revenge-of-the-accountants/overview)         |

examples/bike-sharing-demand.py

@@ -0,0 +1,58 @@
+import pandas as pd
+import numpy as np
+from sklearn.model_selection import train_test_split
+from lightgbm import LGBMRegressor
+from sklearn.metrics import mean_squared_log_error
+
+# Load the data
+train = pd.read_csv("./input/train.csv")
+test = pd.read_csv("./input/test.csv")
+
+
+# Feature engineering
+def preprocess_data(data):
+    data["datetime"] = pd.to_datetime(data["datetime"])
+    data["hour"] = data["datetime"].dt.hour
+    data["day_of_week"] = data["datetime"].dt.dayofweek
+    data["month"] = data["datetime"].dt.month
+    data["year"] = data["datetime"].dt.year
+    data["day"] = data["datetime"].dt.day
+    data["hour_workingday_interaction"] = data["hour"] * data["workingday"]
+
+    # Adding cyclic features
+    data["hour_sin"] = np.sin(data.hour * (2.0 * np.pi / 24))
+    data["hour_cos"] = np.cos(data.hour * (2.0 * np.pi / 24))
+    data["day_of_week_sin"] = np.sin(data.day_of_week * (2.0 * np.pi / 7))
+    data["day_of_week_cos"] = np.cos(data.day_of_week * (2.0 * np.pi / 7))
+    data["month_sin"] = np.sin((data.month - 1) * (2.0 * np.pi / 12))
+    data["month_cos"] = np.cos((data.month - 1) * (2.0 * np.pi / 12))
+
+    return data.drop(["datetime", "casual", "registered"], axis=1, errors="ignore")
+
+
+train = preprocess_data(train)
+test = preprocess_data(test)
+
+# Splitting the training data for validation
+X = train.drop(["count"], axis=1)
+y = np.log1p(train["count"])  # Apply log1p to transform the target variable
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Model training
+model = LGBMRegressor(n_estimators=100, learning_rate=0.05, random_state=42)
+model.fit(X_train, y_train)
+
+# Prediction and evaluation
+y_pred = model.predict(X_val)
+rmsle = np.sqrt(mean_squared_log_error(np.expm1(y_val), np.expm1(y_pred)))
+print(f"RMSLE with cyclic features: {rmsle}")
+
+# Prepare submission
+test_pred = model.predict(test)
+submission = pd.DataFrame(
+    {
+        "datetime": pd.read_csv("./input/test.csv")["datetime"],
+        "count": np.expm1(test_pred),
+    }
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/career-con-2019.py

@@ -0,0 +1,54 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+from sklearn.preprocessing import StandardScaler
+
+# Load the data
+X_train = pd.read_csv("./input/X_train.csv")
+y_train = pd.read_csv("./input/y_train.csv")
+X_test = pd.read_csv("./input/X_test.csv")
+
+# Merge the sensor data with the target variable
+train_data = X_train.merge(y_train, on="series_id", how="inner")
+
+# Drop non-feature columns
+features = train_data.drop(
+    ["row_id", "series_id", "measurement_number", "group_id", "surface"], axis=1
+)
+labels = train_data["surface"]
+
+# Normalize the feature data
+scaler = StandardScaler()
+features = scaler.fit_transform(features)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(
+    features, labels, test_size=0.2, random_state=42
+)
+
+# Initialize the Random Forest classifier
+rf = RandomForestClassifier(n_estimators=100, random_state=42)
+
+# Train the model
+rf.fit(X_train, y_train)
+
+# Predict on the validation set
+y_pred = rf.predict(X_val)
+
+# Calculate the accuracy
+accuracy = accuracy_score(y_val, y_pred)
+print(f"Validation Accuracy: {accuracy}")
+
+# Prepare the test data
+test_features = X_test.drop(["row_id", "series_id", "measurement_number"], axis=1)
+test_features = scaler.transform(test_features)
+
+# Predict on the test set
+test_predictions = rf.predict(test_features)
+
+# Save the predictions to a CSV file
+submission = pd.DataFrame(
+    {"series_id": X_test["series_id"], "surface": test_predictions}
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/cat-in-the-dat-ii.py

@@ -0,0 +1,78 @@
+import pandas as pd
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import roc_auc_score
+from lightgbm import LGBMClassifier
+from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate target from predictors
+y = train_data["target"]
+X = train_data.drop(["target", "id"], axis=1)
+X_test = test_data.drop("id", axis=1)
+
+# List of columns by type
+binary_cols = [col for col in X.columns if "bin" in col]
+ordinal_cols = [col for col in X.columns if "ord" in col]
+nominal_cols = [col for col in X.columns if "nom" in col]
+cyclical_cols = ["day", "month"]
+
+# Ordinal encoding for binary and ordinal features
+ordinal_encoder = OrdinalEncoder()
+X[binary_cols + ordinal_cols] = ordinal_encoder.fit_transform(
+    X[binary_cols + ordinal_cols]
+)
+X_test[binary_cols + ordinal_cols] = ordinal_encoder.transform(
+    X_test[binary_cols + ordinal_cols]
+)
+
+# One-hot encoding for nominal features with low cardinality
+low_cardinality_nom_cols = [col for col in nominal_cols if X[col].nunique() < 10]
+one_hot_encoder = OneHotEncoder(handle_unknown="ignore", sparse=False)
+X_low_card_nom = pd.DataFrame(
+    one_hot_encoder.fit_transform(X[low_cardinality_nom_cols])
+)
+X_test_low_card_nom = pd.DataFrame(
+    one_hot_encoder.transform(X_test[low_cardinality_nom_cols])
+)
+
+# Frequency encoding for nominal features with high cardinality
+high_cardinality_nom_cols = [col for col in nominal_cols if X[col].nunique() >= 10]
+for col in high_cardinality_nom_cols:
+    freq_encoder = X[col].value_counts(normalize=True)
+    X[col] = X[col].map(freq_encoder)
+    X_test[col] = X_test[col].map(freq_encoder)
+
+# Combine all features
+X = pd.concat([X, X_low_card_nom], axis=1).drop(low_cardinality_nom_cols, axis=1)
+X_test = pd.concat([X_test, X_test_low_card_nom], axis=1).drop(
+    low_cardinality_nom_cols, axis=1
+)
+
+# Split the data into training and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Define the model
+model = LGBMClassifier()
+
+# Train the model
+model.fit(X_train, y_train)
+
+# Predict on the validation set
+valid_preds = model.predict_proba(X_valid)[:, 1]
+
+# Evaluate the model
+roc_auc = roc_auc_score(y_valid, valid_preds)
+print(f"Validation ROC AUC Score: {roc_auc}")
+
+# Predict on the test set
+test_preds = model.predict_proba(X_test)[:, 1]
+
+# Save the predictions to a CSV file
+output = pd.DataFrame({"id": test_data.id, "target": test_preds})
+output.to_csv("./working/submission.csv", index=False)

examples/cat-in-the-dat.py

@@ -0,0 +1,58 @@
+import pandas as pd
+from catboost import CatBoostClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["id", "target"], axis=1)
+y = train_data["target"]
+X_test = test_data.drop(["id"], axis=1)
+
+# Identify categorical features
+cat_features = [col for col in X.columns if X[col].dtype == "object"]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize the CatBoostClassifier with a smaller number of iterations for faster grid search
+model = CatBoostClassifier(
+    iterations=100,  # Reduced number of iterations for grid search
+    learning_rate=0.1,
+    depth=4,
+    loss_function="Logloss",
+    early_stopping_rounds=10,
+    verbose=False,
+)
+
+# Fit the model on the training data
+model.fit(X_train, y_train, cat_features=cat_features)
+
+# Predict on the validation set
+val_pred = model.predict_proba(X_val)[:, 1]
+
+# Calculate the ROC AUC score
+val_auc = roc_auc_score(y_val, val_pred)
+print(f"Validation AUC: {val_auc}")
+
+# Train the final model on the full dataset with more iterations
+final_model = CatBoostClassifier(
+    iterations=1000,  # Increased number of iterations for final training
+    learning_rate=0.1,
+    depth=4,
+    loss_function="Logloss",
+    early_stopping_rounds=10,
+    verbose=False,
+)
+
+final_model.fit(X, y, cat_features=cat_features)
+
+# Predict on the test set
+test_pred = final_model.predict_proba(X_test)[:, 1]
+
+# Save the predictions to a CSV file
+submission = pd.DataFrame({"id": test_data["id"], "target": test_pred})
+submission.to_csv("./working/submission.csv", index=False)

examples/ciphertext-challenge-ii.py

@@ -0,0 +1,59 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+import re
+
+# Load the data
+train_df = pd.read_csv("./input/training.csv")
+test_df = pd.read_csv("./input/test.csv")
+
+# Split the training data into training and validation sets
+train, val = train_test_split(train_df, test_size=0.1, random_state=42)
+
+# Ensure the 'ciphertext' column is included in the 'val' dataframe
+val["ciphertext"] = val["text"].apply(lambda x: x)  # Placeholder for actual encryption
+
+
+# Function to perform frequency analysis on a given text
+def frequency_analysis(text):
+    # Remove non-alphabetic characters and convert to uppercase
+    text = re.sub("[^A-Za-z]", "", text).upper()
+    # Count the frequency of each letter in the text
+    return text
+
+
+# Function to decrypt a simple substitution cipher using frequency analysis
+def decrypt_substitution_cipher(ciphertext, frequency_map):
+    # Placeholder for actual decryption
+    return ciphertext
+
+
+# Perform frequency analysis on the validation set plaintext to create a frequency map
+frequency_map = frequency_analysis("".join(val["text"]))
+
+# Decrypt the ciphertext in the validation set and compare with actual plaintext
+val["predicted_text"] = val["ciphertext"].apply(
+    lambda x: decrypt_substitution_cipher(x, frequency_map)
+)
+
+
+# Find the corresponding 'index' from the training set where the decrypted text matches the plaintext
+def find_index(predicted_text, train_df):
+    for index, row in train_df.iterrows():
+        if row["text"] == predicted_text:
+            return row["index"]
+    return None
+
+
+val["predicted_index"] = val["predicted_text"].apply(lambda x: find_index(x, train_df))
+
+# Calculate the accuracy of the predicted index
+accuracy = accuracy_score(val["index"], val["predicted_index"])
+print(f"Validation Accuracy: {accuracy}")
+
+# Decrypt the test set and prepare the submission file
+test_df["predicted_index"] = test_df["ciphertext"].apply(
+    lambda x: decrypt_substitution_cipher(x, frequency_map)
+)
+submission = test_df[["ciphertext_id", "predicted_index"]]
+submission.to_csv("./working/submission.csv", index=False)

examples/ciphertext-challenge-iii.py

@@ -0,0 +1,31 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+
+# Load the data
+train_df = pd.read_csv("./input/train.csv")
+test_df = pd.read_csv("./input/test.csv")
+sample_submission = pd.read_csv("./input/sample_submission.csv")
+
+# Split the training data into training and validation sets
+train_texts, val_texts, train_indices, val_indices = train_test_split(
+    train_df["text"], train_df["index"], test_size=0.1, random_state=42
+)
+
+# Placeholder for the predictions
+val_predictions = [0] * len(val_texts)
+
+# TODO: Implement the decryption algorithm here
+# For now, we are just using a placeholder prediction
+# In a real scenario, this is where the decryption logic would be applied
+
+# Evaluate the accuracy of the predictions
+accuracy = accuracy_score(val_indices, val_predictions)
+print(f"Validation accuracy: {accuracy}")
+
+# Prepare the submission file
+test_predictions = [0] * len(test_df)
+submission = pd.DataFrame(
+    {"ciphertext_id": test_df["ciphertext_id"], "index": test_predictions}
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/competitive-data-science-predict-future-sales.py

@@ -0,0 +1,77 @@
+import pandas as pd
+import numpy as np
+from sklearn.metrics import mean_squared_error
+from lightgbm import LGBMRegressor
+from sklearn.model_selection import train_test_split
+
+# Load data
+sales = pd.read_csv("./input/sales_train.csv")
+test = pd.read_csv("./input/test.csv")
+
+# Convert date to datetime and extract year and month
+sales["date"] = pd.to_datetime(sales["date"], format="%d.%m.%Y")
+sales["year"] = sales["date"].dt.year
+sales["month"] = sales["date"].dt.month
+
+# Aggregate data to monthly level
+monthly_sales = (
+    sales.groupby(["year", "month", "shop_id", "item_id"])
+    .agg({"item_cnt_day": "sum"})
+    .reset_index()
+)
+monthly_sales.rename(columns={"item_cnt_day": "item_cnt_month"}, inplace=True)
+
+# Create lag features
+for lag in [1, 2, 3]:
+    shifted = monthly_sales.copy()
+    shifted["month"] += lag
+    shifted["year"] += shifted["month"] // 12
+    shifted["month"] %= 12
+    shifted.rename(
+        columns={"item_cnt_month": f"item_cnt_month_lag_{lag}"}, inplace=True
+    )
+    monthly_sales = pd.merge(
+        monthly_sales, shifted, on=["year", "month", "shop_id", "item_id"], how="left"
+    )
+
+# Mean encoded features
+item_mean = monthly_sales.groupby("item_id")["item_cnt_month"].mean().reset_index()
+item_mean.rename(columns={"item_cnt_month": "item_mean_cnt"}, inplace=True)
+shop_mean = monthly_sales.groupby("shop_id")["item_cnt_month"].mean().reset_index()
+shop_mean.rename(columns={"item_cnt_month": "shop_mean_cnt"}, inplace=True)
+
+monthly_sales = pd.merge(monthly_sales, item_mean, on="item_id", how="left")
+monthly_sales = pd.merge(monthly_sales, shop_mean, on="shop_id", how="left")
+
+# Prepare training data
+X = monthly_sales.drop(["item_cnt_month", "year", "month"], axis=1)
+y = monthly_sales["item_cnt_month"].clip(0, 20)
+
+# Train/test split
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Model training
+model = LGBMRegressor()
+model.fit(X_train, y_train)
+
+# Predictions
+y_pred = model.predict(X_val).clip(0, 20)
+rmse = np.sqrt(mean_squared_error(y_val, y_pred))
+print(f"Validation RMSE: {rmse}")
+
+# Prepare test set
+test = pd.merge(
+    test,
+    monthly_sales.drop(["item_cnt_month"], axis=1),
+    on=["shop_id", "item_id"],
+    how="left",
+).fillna(0)
+
+# Drop 'year' and 'month' columns to match training data
+test.drop(["year", "month"], axis=1, inplace=True)
+
+# Make predictions on test set
+test["item_cnt_month"] = model.predict(test.drop(["ID"], axis=1)).clip(0, 20)
+
+# Save submission
+test[["ID", "item_cnt_month"]].to_csv("./working/submission.csv", index=False)

examples/digit-recognizer.py

@@ -0,0 +1,79 @@
+import pandas as pd
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from sklearn.model_selection import train_test_split
+from torch.utils.data import DataLoader, TensorDataset
+from torchvision import transforms
+
+# Load the data
+train_df = pd.read_csv("./input/train.csv")
+
+# Prepare the data
+X = train_df.drop("label", axis=1).values.reshape(-1, 1, 28, 28).astype("float32")
+y = train_df["label"].values
+X /= 255.0  # Normalize to [0, 1]
+
+# Split into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Convert to PyTorch tensors
+X_train_tensor = torch.tensor(X_train)
+y_train_tensor = torch.tensor(y_train, dtype=torch.long)
+X_val_tensor = torch.tensor(X_val)
+y_val_tensor = torch.tensor(y_val, dtype=torch.long)
+
+# Create datasets and dataloaders
+train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
+val_dataset = TensorDataset(X_val_tensor, y_val_tensor)
+train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
+val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
+
+
+# Define the CNN model
+class Net(nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, kernel_size=3)
+        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
+        self.fc1 = nn.Linear(64 * 5 * 5, 128)
+        self.fc2 = nn.Linear(128, 10)
+
+    def forward(self, x):
+        x = F.relu(F.max_pool2d(self.conv1(x), 2))
+        x = F.relu(F.max_pool2d(self.conv2(x), 2))
+        x = x.view(-1, 64 * 5 * 5)
+        x = F.relu(self.fc1(x))
+        x = self.fc2(x)
+        return F.log_softmax(x, dim=1)
+
+
+# Initialize the model, loss function, and optimizer
+model = Net()
+criterion = nn.CrossEntropyLoss()
+optimizer = optim.Adam(model.parameters())
+
+# Train the model
+num_epochs = 5
+for epoch in range(num_epochs):
+    model.train()
+    for data, target in train_loader:
+        optimizer.zero_grad()
+        output = model(data)
+        loss = criterion(output, target)
+        loss.backward()
+        optimizer.step()
+
+# Evaluate the model
+model.eval()
+correct = 0
+with torch.no_grad():
+    for data, target in val_loader:
+        output = model(data)
+        pred = output.argmax(dim=1, keepdim=True)
+        correct += pred.eq(target.view_as(pred)).sum().item()
+
+accuracy = correct / len(val_loader.dataset)
+print(f"Validation Accuracy: {accuracy:.4f}")

examples/dont-overfit-ii.py

@@ -0,0 +1,45 @@
+import pandas as pd
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import StratifiedKFold
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+X_train = train_data.drop(["id", "target"], axis=1)
+y_train = train_data["target"]
+
+# Initialize the model with L1 regularization
+model = LogisticRegression(penalty="l1", solver="liblinear", random_state=42)
+
+# Prepare cross-validation
+cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)
+auc_scores = []
+
+# Perform 10-fold cross-validation
+for train_idx, valid_idx in cv.split(X_train, y_train):
+    X_train_fold, X_valid_fold = X_train.iloc[train_idx], X_train.iloc[valid_idx]
+    y_train_fold, y_valid_fold = y_train.iloc[train_idx], y_train.iloc[valid_idx]
+
+    # Train the model
+    model.fit(X_train_fold, y_train_fold)
+
+    # Predict probabilities for the validation set
+    y_pred_prob = model.predict_proba(X_valid_fold)[:, 1]
+
+    # Calculate the AUC score and append to the list
+    auc_score = roc_auc_score(y_valid_fold, y_pred_prob)
+    auc_scores.append(auc_score)
+
+# Calculate the average AUC score across all folds
+average_auc_score = sum(auc_scores) / len(auc_scores)
+print(f"Average AUC-ROC score: {average_auc_score}")
+
+# Train the model on the full training set and predict for the test set
+model.fit(X_train, y_train)
+test_data = pd.read_csv("./input/test.csv")
+X_test = test_data.drop("id", axis=1)
+test_data["target"] = model.predict_proba(X_test)[:, 1]
+
+# Save the submission file
+submission_file = "./working/submission.csv"
+test_data[["id", "target"]].to_csv(submission_file, index=False)

examples/facial-keypoints-detection.py

@@ -0,0 +1,106 @@
+import pandas as pd
+import numpy as np
+import torch
+import torch.nn as nn
+from torch.utils.data import Dataset, DataLoader
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error
+
+# Load the data
+train_df = pd.read_csv("./input/training/training.csv")
+train_df.dropna(inplace=True)  # Remove missing values for simplicity
+
+# Preprocess the data
+X = (
+    np.vstack(train_df["Image"].apply(lambda x: np.fromstring(x, sep=" ")).values)
+    / 255.0
+)  # Normalize pixel values
+X = X.reshape(-1, 96, 96, 1)
+y = train_df.drop(["Image"], axis=1).values
+
+# Train-test split
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+
+# Define dataset
+class FacesDataset(Dataset):
+    def __init__(self, images, keypoints):
+        self.images = images
+        self.keypoints = keypoints
+
+    def __len__(self):
+        return len(self.images)
+
+    def __getitem__(self, idx):
+        image = torch.tensor(self.images[idx], dtype=torch.float32).permute(2, 0, 1)
+        keypoint = torch.tensor(self.keypoints[idx], dtype=torch.float32)
+        return image, keypoint
+
+
+# Define model
+class KeypointModel(nn.Module):
+    def __init__(self):
+        super(KeypointModel, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)
+        self.pool = nn.MaxPool2d(2, 2)
+        self.fc1 = nn.Linear(32 * 48 * 48, 1000)
+        self.fc2 = nn.Linear(1000, 30)
+
+    def forward(self, x):
+        x = self.pool(torch.relu(self.conv1(x)))
+        x = x.view(x.size(0), -1)
+        x = torch.relu(self.fc1(x))
+        x = self.fc2(x)
+        return x
+
+
+# Training
+def train(model, criterion, optimizer, train_loader, val_loader, epochs=10):
+    for epoch in range(epochs):
+        model.train()
+        running_loss = 0.0
+        for images, keypoints in train_loader:
+            optimizer.zero_grad()
+            outputs = model(images)
+            loss = criterion(outputs, keypoints)
+            loss.backward()
+            optimizer.step()
+            running_loss += loss.item()
+
+        model.eval()
+        val_loss = 0.0
+        with torch.no_grad():
+            for images, keypoints in val_loader:
+                outputs = model(images)
+                loss = criterion(outputs, keypoints)
+                val_loss += loss.item()
+
+        print(
+            f"Epoch {epoch+1}, Train Loss: {running_loss/len(train_loader)}, Val Loss: {val_loss/len(val_loader)}"
+        )
+
+
+# Initialize dataset, model, criterion, and optimizer
+train_dataset = FacesDataset(X_train, y_train)
+val_dataset = FacesDataset(X_val, y_val)
+train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
+val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
+model = KeypointModel()
+criterion = nn.MSELoss()
+optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
+
+# Train the model
+train(model, criterion, optimizer, train_loader, val_loader, epochs=10)
+
+# Evaluation
+model.eval()
+predictions = []
+ground_truths = []
+with torch.no_grad():
+    for images, keypoints in val_loader:
+        outputs = model(images)
+        predictions.extend(outputs.numpy())
+        ground_truths.extend(keypoints.numpy())
+
+rmse = np.sqrt(mean_squared_error(ground_truths, predictions))
+print(f"Validation RMSE: {rmse}")

examples/forest-cover-type-prediction.py

@@ -0,0 +1,33 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["Id", "Cover_Type"], axis=1)
+y = train_data["Cover_Type"]
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize the model
+rf = RandomForestClassifier(n_estimators=100, random_state=42)
+
+# Train the model
+rf.fit(X_train, y_train)
+
+# Validate the model
+y_pred = rf.predict(X_val)
+accuracy = accuracy_score(y_val, y_pred)
+print(f"Validation Accuracy: {accuracy}")
+
+# Predict on test data
+test_ids = test_data["Id"]
+test_data = test_data.drop("Id", axis=1)
+test_predictions = rf.predict(test_data)
+
+# Save the predictions
+submission = pd.DataFrame({"Id": test_ids, "Cover_Type": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/godaddy-microbusiness-density-forecasting.py

@@ -0,0 +1,77 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.model_selection import cross_val_score, KFold, GridSearchCV
+from sklearn.metrics import make_scorer
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+census_data = pd.read_csv("./input/census_starter.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Merge train and test data with census data
+train_data = train_data.merge(census_data, on="cfips", how="left")
+test_data = test_data.merge(census_data, on="cfips", how="left")
+
+# Preprocessing for numerical data
+numerical_transformer = SimpleImputer(strategy="median")
+
+# Columns to be used as features
+feature_columns = train_data.select_dtypes(exclude=["object", "datetime"]).columns.drop(
+    "microbusiness_density"
+)
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, feature_columns),
+    ]
+)
+
+# Define the model
+model = RandomForestRegressor(random_state=0)
+
+# Bundle preprocessing and modeling code in a pipeline
+my_pipeline = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+
+# Define SMAPE function
+def smape(actual, predicted):
+    denominator = (abs(actual) + abs(predicted)) / 2.0
+    diff = abs(predicted - actual) / denominator
+    diff[denominator == 0] = 0.0
+    return 100 * diff.mean()
+
+
+smape_scorer = make_scorer(smape, greater_is_better=False)
+
+# Define the grid of hyperparameters to search
+param_grid = {
+    "model__n_estimators": [50, 100, 150],
+    "model__max_depth": [None, 10, 20, 30],
+    "model__min_samples_split": [2, 5, 10],
+}
+
+# Set up GridSearchCV
+grid_search = GridSearchCV(
+    my_pipeline, param_grid=param_grid, cv=3, scoring=smape_scorer, n_jobs=-1
+)
+
+# Fit the grid search to the data
+grid_search.fit(train_data[feature_columns], train_data["microbusiness_density"])
+
+# Print the best parameters and the corresponding SMAPE score
+print(f"Best parameters: {grid_search.best_params_}")
+print(f"Best SMAPE score: {-grid_search.best_score_}")
+
+# Fit the model with the best parameters and make predictions on the test set
+best_pipeline = grid_search.best_estimator_
+best_pipeline.fit(train_data[feature_columns], train_data["microbusiness_density"])
+test_preds = best_pipeline.predict(test_data[feature_columns])
+
+# Save test predictions to file
+output = pd.DataFrame({"row_id": test_data.row_id, "microbusiness_density": test_preds})
+output.to_csv("./working/submission.csv", index=False)

examples/home-data-for-ml-course.py

@@ -0,0 +1,64 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.metrics import mean_squared_error
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate target from predictors
+y = train_data.SalePrice
+X = train_data.drop(["SalePrice"], axis=1)
+
+# Preprocessing for numerical data
+numerical_transformer = SimpleImputer(strategy="median")
+
+# Preprocessing for categorical data
+categorical_transformer = Pipeline(
+    steps=[
+        ("imputer", SimpleImputer(strategy="most_frequent")),
+        ("onehot", OneHotEncoder(handle_unknown="ignore")),
+    ]
+)
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, X.select_dtypes(exclude=["object"]).columns),
+        ("cat", categorical_transformer, X.select_dtypes(include=["object"]).columns),
+    ]
+)
+
+# Define the model
+model = GradientBoostingRegressor()
+
+# Bundle preprocessing and modeling code in a pipeline
+my_pipeline = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Split data into training and validation subsets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Preprocessing of training data, fit model
+my_pipeline.fit(X_train, np.log(y_train))
+
+# Preprocessing of validation data, get predictions
+preds = my_pipeline.predict(X_valid)
+
+# Evaluate the model
+score = mean_squared_error(np.log(y_valid), preds, squared=False)
+print("RMSE:", score)
+
+# Preprocessing of test data, fit model
+test_preds = my_pipeline.predict(test_data)
+
+# Save test predictions to file
+output = pd.DataFrame({"Id": test_data.Id, "SalePrice": np.exp(test_preds)})
+output.to_csv("./working/submission.csv", index=False)

examples/house-prices-advanced-regression-techniques.py

@@ -0,0 +1,79 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.linear_model import Lasso
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.metrics import mean_squared_error
+import numpy as np
+
+# Load the data
+train = pd.read_csv("./input/train.csv")
+test = pd.read_csv("./input/test.csv")
+
+# Identify key features for interaction terms based on domain knowledge
+key_features = ["OverallQual", "GrLivArea", "TotalBsmtSF", "GarageCars"]
+
+# Create interaction terms for both train and test datasets
+for i in range(len(key_features)):
+    for j in range(i + 1, len(key_features)):
+        name = key_features[i] + "_X_" + key_features[j]
+        train[name] = train[key_features[i]] * train[key_features[j]]
+        test[name] = test[key_features[i]] * test[key_features[j]]
+
+# Separate features and target variable
+X = train.drop(["SalePrice", "Id"], axis=1)
+y = np.log(train["SalePrice"])  # Log transformation
+test_ids = test["Id"]
+test = test.drop(["Id"], axis=1)
+
+# Preprocessing for numerical data
+numerical_transformer = Pipeline(
+    steps=[("imputer", SimpleImputer(strategy="median")), ("scaler", StandardScaler())]
+)
+
+# Preprocessing for categorical data
+categorical_transformer = Pipeline(
+    steps=[
+        ("imputer", SimpleImputer(strategy="most_frequent")),
+        ("onehot", OneHotEncoder(handle_unknown="ignore")),
+    ]
+)
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, X.select_dtypes(exclude=["object"]).columns),
+        ("cat", categorical_transformer, X.select_dtypes(include=["object"]).columns),
+    ]
+)
+
+# Define model
+model = Pipeline(
+    steps=[("preprocessor", preprocessor), ("regressor", Lasso(alpha=0.001))]
+)
+
+# Split data into training and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Train the model
+model.fit(X_train, y_train)
+
+# Predict on validation set
+preds_valid = model.predict(X_valid)
+
+# Evaluate the model
+score = mean_squared_error(y_valid, preds_valid, squared=False)
+print(f"Validation RMSE: {score}")
+
+# Predict on test data
+test_preds = model.predict(test)
+
+# Save test predictions to file
+output = pd.DataFrame(
+    {"Id": test_ids, "SalePrice": np.exp(test_preds)}
+)  # Re-transform to original scale
+output.to_csv("./working/submission.csv", index=False)

examples/icr-identify-age-related-conditions.py

@@ -0,0 +1,66 @@
+import pandas as pd
+import lightgbm as lgb
+from sklearn.model_selection import RandomizedSearchCV, KFold
+from sklearn.metrics import log_loss, make_scorer
+from sklearn.preprocessing import LabelEncoder
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Encode categorical features
+le = LabelEncoder()
+train_data["EJ"] = le.fit_transform(train_data["EJ"])
+test_data["EJ"] = le.transform(test_data["EJ"])
+
+# Prepare the data
+X = train_data.drop(["Id", "Class"], axis=1)
+y = train_data["Class"]
+X_test = test_data.drop("Id", axis=1)
+
+# Define the model parameters and parameter grid for randomized search
+model = lgb.LGBMClassifier(objective="binary", boosting_type="gbdt", is_unbalance=True)
+param_grid = {
+    "learning_rate": [0.01, 0.05, 0.1],
+    "num_leaves": [15, 31, 63],
+    "max_depth": [-1, 5, 10],
+    "min_child_samples": [10, 20, 30],
+    "max_bin": [255, 300],
+    "subsample": [0.6, 0.8, 1.0],
+    "colsample_bytree": [0.3, 0.5, 0.7],
+}
+
+# Create a scorer for log loss
+log_loss_scorer = make_scorer(log_loss, greater_is_better=False, needs_proba=True)
+
+# Perform randomized search with cross-validation
+random_search = RandomizedSearchCV(
+    model,
+    param_distributions=param_grid,
+    n_iter=10,
+    scoring=log_loss_scorer,
+    cv=KFold(n_splits=10, shuffle=True, random_state=42),
+    random_state=42,
+    verbose=1,
+)
+
+random_search.fit(X, y)
+
+# Best model and log loss
+best_model = random_search.best_estimator_
+best_score = -random_search.best_score_
+print(f"Best Log Loss: {best_score}")
+
+# Predict on test set with the best model
+test_predictions = best_model.predict_proba(X_test)[:, 1]
+
+# Create a submission file
+submission = pd.DataFrame(
+    {
+        "Id": test_data["Id"],
+        "class_0": 1 - test_predictions,
+        "class_1": test_predictions,
+    }
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/jigsaw-toxic-comment-classification-challenge.py

@@ -0,0 +1,34 @@
+import pandas as pd
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+
+# Prepare the features and labels
+X = train_data["comment_text"]
+y = train_data.iloc[:, 2:]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Create TF-IDF features
+tfidf_vectorizer = TfidfVectorizer(max_features=10000, stop_words="english")
+X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)
+X_val_tfidf = tfidf_vectorizer.transform(X_val)
+
+# Train a logistic regression model for each label
+scores = []
+for label in y.columns:
+    lr = LogisticRegression(C=1.0, solver="liblinear")
+    lr.fit(X_train_tfidf, y_train[label])
+    y_pred = lr.predict_proba(X_val_tfidf)[:, 1]
+    score = roc_auc_score(y_val[label], y_pred)
+    scores.append(score)
+    print(f"ROC AUC for {label}: {score}")
+
+# Calculate the mean column-wise ROC AUC
+mean_auc = sum(scores) / len(scores)
+print(f"Mean column-wise ROC AUC: {mean_auc}")

examples/new-york-city-taxi-fare-prediction.py

@@ -0,0 +1,116 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.metrics import mean_squared_error
+from sklearn.model_selection import train_test_split
+import numpy as np
+
+# Load a subset of the training data
+train_df = pd.read_csv("./input/train.csv", nrows=500000)
+
+# Remove missing values and outliers
+train_df = train_df.dropna(how="any", axis="rows")
+train_df = train_df[(train_df.fare_amount >= 2.5) & (train_df.fare_amount <= 500)]
+train_df = train_df[(train_df.passenger_count > 0) & (train_df.passenger_count <= 6)]
+train_df = train_df[
+    (train_df["pickup_latitude"] != 0) | (train_df["pickup_longitude"] != 0)
+]
+train_df = train_df[
+    (train_df["dropoff_latitude"] != 0) | (train_df["dropoff_longitude"] != 0)
+]
+
+
+# Feature engineering
+def haversine_distance(lat1, lon1, lat2, lon2):
+    R = 6371  # radius of Earth in kilometers
+    phi1 = np.radians(lat1)
+    phi2 = np.radians(lat2)
+    delta_phi = np.radians(lat2 - lat1)
+    delta_lambda = np.radians(lon2 - lon1)
+    a = (
+        np.sin(delta_phi / 2) ** 2
+        + np.cos(phi1) * np.cos(phi2) * np.sin(delta_lambda / 2) ** 2
+    )
+    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1 - a))
+    d = R * c
+    return d
+
+
+train_df["pickup_datetime"] = pd.to_datetime(train_df["pickup_datetime"])
+train_df["year"] = train_df["pickup_datetime"].dt.year
+train_df["month"] = train_df["pickup_datetime"].dt.month
+train_df["day"] = train_df["pickup_datetime"].dt.day
+train_df["hour"] = train_df["pickup_datetime"].dt.hour
+train_df["weekday"] = train_df["pickup_datetime"].dt.weekday
+train_df["distance"] = haversine_distance(
+    train_df["pickup_latitude"],
+    train_df["pickup_longitude"],
+    train_df["dropoff_latitude"],
+    train_df["dropoff_longitude"],
+)
+
+# Select features and target variable
+features = [
+    "year",
+    "month",
+    "day",
+    "hour",
+    "weekday",
+    "passenger_count",
+    "pickup_latitude",
+    "pickup_longitude",
+    "dropoff_latitude",
+    "dropoff_longitude",
+    "distance",
+]
+target = "fare_amount"
+
+X = train_df[features]
+y = train_df[target]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Train the model
+rf = RandomForestRegressor(n_estimators=50, max_depth=25, random_state=42)
+rf.fit(X_train, y_train)
+
+# Predict on validation set
+y_pred = rf.predict(X_val)
+
+# Calculate RMSE
+rmse = np.sqrt(mean_squared_error(y_val, y_pred))
+print(f"Validation RMSE: {rmse}")
+
+# Prepare the test set
+test_df = pd.read_csv("./input/test.csv")
+test_df["pickup_datetime"] = pd.to_datetime(test_df["pickup_datetime"])
+test_df["year"] = test_df["pickup_datetime"].dt.year
+test_df["month"] = test_df["pickup_datetime"].dt.month
+test_df["day"] = test_df["pickup_datetime"].dt.day
+test_df["hour"] = test_df["pickup_datetime"].dt.hour
+test_df["weekday"] = test_df["pickup_datetime"].dt.weekday
+
+# Impute NaN values in the test set using median from the training set
+for feature in [
+    "pickup_latitude",
+    "pickup_longitude",
+    "dropoff_latitude",
+    "dropoff_longitude",
+]:
+    median_value = train_df[feature].median()
+    test_df[feature].fillna(median_value, inplace=True)
+
+test_df["distance"] = haversine_distance(
+    test_df["pickup_latitude"],
+    test_df["pickup_longitude"],
+    test_df["dropoff_latitude"],
+    test_df["dropoff_longitude"],
+)
+
+# Predict on test set
+X_test = test_df[features]
+test_df["fare_amount"] = rf.predict(X_test)
+
+# Save predictions
+submission = test_df[["key", "fare_amount"]]
+submission.to_csv("./working/submission.csv", index=False)

examples/nlp-getting-started.py

@@ -0,0 +1,36 @@
+import pandas as pd
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import f1_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X_train, X_val, y_train, y_val = train_test_split(
+    train_data["text"], train_data["target"], test_size=0.2, random_state=42
+)
+
+# Vectorize the text data
+vectorizer = TfidfVectorizer()
+X_train_tfidf = vectorizer.fit_transform(X_train)
+X_val_tfidf = vectorizer.transform(X_val)
+
+# Train the logistic regression model
+model = LogisticRegression()
+model.fit(X_train_tfidf, y_train)
+
+# Predict on the validation set
+val_predictions = model.predict(X_val_tfidf)
+
+# Evaluate the model
+f1 = f1_score(y_val, val_predictions)
+print(f"F1 Score on the validation set: {f1}")
+
+# Predict on the test set and save the submission
+X_test_tfidf = vectorizer.transform(test_data["text"])
+test_predictions = model.predict(X_test_tfidf)
+submission = pd.DataFrame({"id": test_data["id"], "target": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/optiver-trading-at-the-close.py

@@ -0,0 +1,58 @@
+import pandas as pd
+import numpy as np
+from lightgbm import LGBMRegressor
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_absolute_error
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Preprocess the data: fill missing values with median for numeric columns only
+numeric_columns_train = train_data.select_dtypes(include=[np.number]).columns
+train_data[numeric_columns_train] = train_data[numeric_columns_train].fillna(
+    train_data[numeric_columns_train].median()
+)
+
+# Ensure 'target' is not in the numeric columns for test data
+numeric_columns_test = test_data.select_dtypes(include=[np.number]).columns
+test_data[numeric_columns_test] = test_data[numeric_columns_test].fillna(
+    test_data[numeric_columns_test].median()
+)
+
+# Prepare features and target
+X = train_data.drop(["row_id", "target"], axis=1)
+y = train_data["target"]
+
+# Initialize LightGBM regressor
+model = LGBMRegressor()
+
+# Prepare cross-validation
+kf = KFold(n_splits=10, shuffle=True, random_state=42)
+mae_scores = []
+
+# Perform 10-fold cross-validation
+for train_index, val_index in kf.split(X):
+    X_train, X_val = X.iloc[train_index], X.iloc[val_index]
+    y_train, y_val = y.iloc[train_index], y.iloc[val_index]
+
+    # Train the model
+    model.fit(X_train, y_train)
+
+    # Predict on validation set
+    y_pred = model.predict(X_val)
+
+    # Calculate and store MAE
+    mae = mean_absolute_error(y_val, y_pred)
+    mae_scores.append(mae)
+
+# Print the average MAE across all folds
+print(f"Average MAE: {np.mean(mae_scores)}")
+
+# Predict on test set
+test_features = test_data.drop(["row_id"], axis=1)
+test_data["target"] = model.predict(test_features)
+
+# Save predictions to submission.csv
+submission = test_data[["row_id", "target"]]
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e1.py

@@ -0,0 +1,55 @@
+import lightgbm as lgb
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+sample_submission = pd.read_csv("./input/sample_submission.csv")
+
+# Prepare the data
+X = train_data.drop(["MedHouseVal", "id"], axis=1)
+y = train_data["MedHouseVal"]
+X_test = test_data.drop("id", axis=1)
+
+# Prepare cross-validation
+kf = KFold(n_splits=10, shuffle=True, random_state=42)
+rmse_scores = []
+
+# Perform 10-fold cross-validation
+for train_index, val_index in kf.split(X):
+    X_train, X_val = X.iloc[train_index], X.iloc[val_index]
+    y_train, y_val = y.iloc[train_index], y.iloc[val_index]
+
+    # Create LightGBM datasets
+    train_set = lgb.Dataset(X_train, label=y_train)
+    val_set = lgb.Dataset(X_val, label=y_val)
+
+    # Train the model
+    params = {"objective": "regression", "metric": "rmse", "verbosity": -1}
+    model = lgb.train(
+        params, train_set, valid_sets=[train_set, val_set], verbose_eval=False
+    )
+
+    # Predict on validation set
+    y_pred = model.predict(X_val, num_iteration=model.best_iteration)
+
+    # Calculate RMSE
+    rmse = np.sqrt(mean_squared_error(y_val, y_pred))
+    rmse_scores.append(rmse)
+
+# Print the average RMSE across the folds
+print(f"Average RMSE: {np.mean(rmse_scores)}")
+
+# Train the model on the full dataset
+full_train_set = lgb.Dataset(X, label=y)
+final_model = lgb.train(params, full_train_set, verbose_eval=False)
+
+# Predict on the test set
+predictions = final_model.predict(X_test, num_iteration=final_model.best_iteration)
+
+# Prepare the submission file
+submission = pd.DataFrame({"id": test_data["id"], "MedHouseVal": predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e11.py

@@ -0,0 +1,33 @@
+import pandas as pd
+import lightgbm as lgb
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_log_error
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "cost"], axis=1)
+y = train_data["cost"]
+X_test = test_data.drop("id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Train the model
+model = lgb.LGBMRegressor(random_state=42)
+model.fit(X_train, y_train)
+
+# Make predictions
+y_pred = model.predict(X_val)
+y_pred_test = model.predict(X_test)
+
+# Calculate the RMSLE
+rmsle = np.sqrt(mean_squared_log_error(y_val, y_pred))
+print(f"Validation RMSLE: {rmsle}")
+
+# Prepare the submission file
+submission = pd.DataFrame({"id": test_data["id"], "cost": y_pred_test})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e13.py

@@ -0,0 +1,69 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.multiclass import OneVsRestClassifier
+from sklearn.preprocessing import LabelEncoder
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "prognosis"], axis=1)
+y = train_data["prognosis"]
+X_test = test_data.drop(["id"], axis=1)
+test_ids = test_data["id"]
+
+# Encode the target variable
+label_encoder = LabelEncoder()
+y_encoded = label_encoder.fit_transform(y)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(
+    X, y_encoded, test_size=0.2, random_state=42
+)
+
+# Train the model
+model = OneVsRestClassifier(RandomForestClassifier(n_estimators=100, random_state=42))
+model.fit(X_train, y_train)
+
+# Predict on the validation set
+y_val_pred_proba = model.predict_proba(X_val)
+
+# Select the top 3 predictions for each sample
+top3_preds = pd.DataFrame(y_val_pred_proba).apply(
+    lambda x: label_encoder.inverse_transform(x.argsort()[-3:][::-1]), axis=1
+)
+
+
+# Evaluate the model using MPA@3
+def mpa_at_k(y_true, y_pred, k=3):
+    score = 0.0
+    for true, pred in zip(y_true, y_pred):
+        try:
+            index = list(pred).index(true)
+            score += 1.0 / (index + 1)
+        except ValueError:
+            continue
+    return score / len(y_true)
+
+
+# Calculate the MPA@3 score
+y_val_true = label_encoder.inverse_transform(y_val)
+mpa_score = mpa_at_k(y_val_true, top3_preds)
+print(f"MPA@3 score on the validation set: {mpa_score}")
+
+# Predict on the test set
+y_test_pred_proba = model.predict_proba(X_test)
+top3_test_preds = pd.DataFrame(y_test_pred_proba).apply(
+    lambda x: label_encoder.inverse_transform(x.argsort()[-3:][::-1]), axis=1
+)
+
+# Prepare the submission file
+submission = pd.DataFrame(
+    {
+        "id": test_ids,
+        "prognosis": [" ".join(map(str, preds)) for preds in top3_test_preds],
+    }
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e14.py

@@ -0,0 +1,61 @@
+import pandas as pd
+from sklearn.ensemble import (
+    GradientBoostingRegressor,
+    RandomForestRegressor,
+    StackingRegressor,
+)
+from sklearn.linear_model import LinearRegression, RidgeCV
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_absolute_error
+from sklearn.preprocessing import StandardScaler
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "yield"], axis=1)
+y = train_data["yield"]
+X_test = test_data.drop("id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Scale the features
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train)
+X_val_scaled = scaler.transform(X_val)
+X_test_scaled = scaler.transform(X_test)
+
+# Initialize base models
+estimators = [
+    (
+        "gbr",
+        GradientBoostingRegressor(
+            n_estimators=200, learning_rate=0.1, max_depth=4, random_state=42
+        ),
+    ),
+    ("rf", RandomForestRegressor(n_estimators=200, random_state=42)),
+    ("lr", LinearRegression()),
+]
+
+# Initialize the StackingRegressor with a RidgeCV final estimator
+stacking_regressor = StackingRegressor(estimators=estimators, final_estimator=RidgeCV())
+
+# Train the StackingRegressor on the scaled training data
+stacking_regressor.fit(X_train_scaled, y_train)
+
+# Predict on the scaled validation set using the StackingRegressor
+y_val_pred = stacking_regressor.predict(X_val_scaled)
+
+# Evaluate the model
+mae = mean_absolute_error(y_val, y_val_pred)
+print(f"Mean Absolute Error on validation set with StackingRegressor: {mae}")
+
+# Train the StackingRegressor on the full scaled training data and predict on the scaled test set
+stacking_regressor.fit(scaler.transform(X), y)
+test_predictions = stacking_regressor.predict(X_test_scaled)
+
+# Save the predictions to a CSV file
+submission = pd.DataFrame({"id": test_data["id"], "yield": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e15.py

@@ -0,0 +1,51 @@
+import pandas as pd
+from sklearn.linear_model import LinearRegression
+from sklearn.model_selection import cross_val_score, KFold
+from sklearn.impute import SimpleImputer
+import numpy as np
+
+# Load the data
+data = pd.read_csv("./input/data.csv")
+
+# Select only numeric columns for imputation
+numeric_columns = data.select_dtypes(include=[np.number]).columns
+X = data[numeric_columns].drop(columns=["x_e_out [-]"])
+y = data["x_e_out [-]"]
+
+# Split the data into training and NaN sets
+train_data = data.dropna(subset=["x_e_out [-]"])
+nan_data = data[data["x_e_out [-]"].isna()]
+
+# Impute missing values in features with mean
+imputer = SimpleImputer(strategy="mean")
+X_train_imputed = imputer.fit_transform(
+    train_data[numeric_columns].drop(columns=["x_e_out [-]"])
+)
+y_train = train_data["x_e_out [-]"]
+
+# Train the linear regression model using cross-validation
+model = LinearRegression()
+kf = KFold(n_splits=10, shuffle=True, random_state=1)
+rmse_scores = cross_val_score(
+    model, X_train_imputed, y_train, scoring="neg_root_mean_squared_error", cv=kf
+)
+print(f"10-fold CV RMSE: {-np.mean(rmse_scores):.4f} (+/- {np.std(rmse_scores):.4f})")
+
+# Fit the model on the entire training set
+model.fit(X_train_imputed, y_train)
+
+# Load the test data
+test_data = pd.read_csv("./input/sample_submission.csv")
+
+# Prepare the test features
+X_test = nan_data[numeric_columns].drop(columns=["x_e_out [-]"])
+X_test_imputed = imputer.transform(X_test)
+
+# Predict the missing values for the test set
+nan_data["x_e_out [-]"] = model.predict(X_test_imputed)
+
+# Merge predictions back into the test data
+test_data = test_data.merge(nan_data[["id", "x_e_out [-]"]], on="id", how="left")
+
+# Save the submission file
+test_data.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e16.py

@@ -0,0 +1,90 @@
+import pandas as pd
+from catboost import CatBoostRegressor
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_absolute_error
+from sklearn.preprocessing import PolynomialFeatures
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["Age", "id"], axis=1)
+y = train_data["Age"]
+test_X = test_data.drop(["id"], axis=1)
+
+
+# Generate polynomial features for selected columns and ensure unique feature names by adding a prefix
+def generate_poly_features(
+    df, feature_names, degree=2, include_bias=False, prefix="poly_"
+):
+    poly_features = PolynomialFeatures(degree=degree, include_bias=include_bias)
+    selected_features = df[feature_names]
+    poly_features_array = poly_features.fit_transform(selected_features)
+    poly_feature_names = [
+        prefix + name for name in poly_features.get_feature_names_out(feature_names)
+    ]
+    return pd.DataFrame(poly_features_array, columns=poly_feature_names)
+
+
+# Apply polynomial feature generation to both train and test datasets
+poly_features_train = generate_poly_features(X, ["Length", "Diameter", "Height"])
+poly_features_test = generate_poly_features(test_X, ["Length", "Diameter", "Height"])
+
+# Concatenate the polynomial features with the original dataset
+X_poly = pd.concat([X.reset_index(drop=True), poly_features_train], axis=1)
+test_X_poly = pd.concat([test_X.reset_index(drop=True), poly_features_test], axis=1)
+
+# Specify categorical features
+cat_features = ["Sex"]
+
+# Initialize KFold
+kf = KFold(n_splits=10, shuffle=True, random_state=42)
+
+# Initialize an empty list to store MAE for each fold
+mae_scores = []
+
+# Define hyperparameters
+hyperparams = {
+    "iterations": 1500,
+    "learning_rate": 0.05,
+    "depth": 8,
+    "loss_function": "MAE",
+    "cat_features": cat_features,
+    "verbose": 0,
+}
+
+# Loop over each fold
+for train_index, test_index in kf.split(X_poly):
+    X_train, X_val = X_poly.iloc[train_index], X_poly.iloc[test_index]
+    y_train, y_val = y.iloc[train_index], y.iloc[test_index]
+
+    # Initialize CatBoostRegressor with hyperparameters
+    model = CatBoostRegressor(**hyperparams)
+
+    # Train the model
+    model.fit(
+        X_train,
+        y_train,
+        cat_features=cat_features,
+        eval_set=(X_val, y_val),
+        early_stopping_rounds=100,
+        verbose=0,
+    )
+
+    # Predict on validation set
+    predictions = model.predict(X_val)
+
+    # Calculate and print MAE
+    mae = mean_absolute_error(y_val, predictions)
+    mae_scores.append(mae)
+
+# Print the average MAE across all folds
+print(f"Average MAE across all folds: {sum(mae_scores) / len(mae_scores)}")
+
+# Predict on the test set
+test_predictions = model.predict(test_X_poly)
+
+# Prepare submission file
+submission_df = pd.DataFrame({"id": test_data["id"], "Age": test_predictions})
+submission_df.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e17.py

@@ -0,0 +1,66 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split, GridSearchCV
+from sklearn.metrics import roc_auc_score
+from sklearn.preprocessing import OneHotEncoder
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# One-hot encode categorical features
+encoder = OneHotEncoder(sparse=False, handle_unknown="ignore")
+encoded_features = encoder.fit_transform(train_data[["Product ID", "Type"]])
+encoded_test_features = encoder.transform(test_data[["Product ID", "Type"]])
+
+# Add encoded features back to the dataframe
+encoded_df = pd.DataFrame(encoded_features, columns=encoder.get_feature_names_out())
+train_data = train_data.join(encoded_df).drop(["Product ID", "Type"], axis=1)
+
+encoded_test_df = pd.DataFrame(
+    encoded_test_features, columns=encoder.get_feature_names_out()
+)
+test_data = test_data.join(encoded_test_df).drop(["Product ID", "Type"], axis=1)
+
+# Split the data into features and target
+X = train_data.drop(["Machine failure", "id"], axis=1)
+y = train_data["Machine failure"]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize the Random Forest classifier
+rf = RandomForestClassifier(random_state=42)
+
+# Define the parameter grid
+param_grid = {
+    "n_estimators": [100, 200],
+    "max_depth": [None, 10, 20],
+    "min_samples_split": [2, 5],
+    "min_samples_leaf": [1, 2],
+}
+
+# Initialize GridSearchCV
+grid_search = GridSearchCV(
+    estimator=rf, param_grid=param_grid, cv=3, scoring="roc_auc", n_jobs=-1
+)
+
+# Perform grid search
+grid_search.fit(X_train, y_train)
+
+# Get the best estimator
+best_rf = grid_search.best_estimator_
+
+# Predict on the validation set using the best estimator
+y_pred_proba = best_rf.predict_proba(X_val)[:, 1]
+
+# Calculate the AUC-ROC score
+auc_roc = roc_auc_score(y_val, y_pred_proba)
+print(f"AUC-ROC score: {auc_roc}")
+
+# Predict on the test set using the best estimator
+test_predictions = best_rf.predict_proba(test_data.drop("id", axis=1))[:, 1]
+
+# Create the submission file
+submission = pd.DataFrame({"id": test_data["id"], "Machine failure": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e18.py

@@ -0,0 +1,91 @@
+import pandas as pd
+import numpy as np
+from lightgbm import LGBMClassifier
+from sklearn.ensemble import (
+    RandomForestClassifier,
+    VotingClassifier,
+    GradientBoostingClassifier,
+)
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import cross_val_score, StratifiedKFold, GridSearchCV
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Identify common features
+common_features = list(set(train_data.columns) & set(test_data.columns))
+common_features.remove("id")
+
+# Prepare the data
+X_train = train_data[common_features]
+y_train_EC1 = train_data["EC1"]
+y_train_EC2 = train_data["EC2"]
+X_test = test_data[common_features]
+
+# Initialize StratifiedKFold for cross-validation
+skf = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)
+
+# Initialize individual models
+model1_EC1 = LGBMClassifier(random_state=42)
+model1_EC2 = LGBMClassifier(random_state=42)
+model2 = RandomForestClassifier(n_estimators=100, random_state=42)
+model3 = LogisticRegression(max_iter=1000)
+model4 = GradientBoostingClassifier(random_state=42)
+
+# Define the parameter grid for GradientBoostingClassifier
+param_grid = {
+    "n_estimators": [100, 200],
+    "learning_rate": [0.05, 0.1],
+    "max_depth": [3, 5],
+}
+
+# Perform GridSearchCV to find the best parameters for GradientBoostingClassifier
+grid_search = GridSearchCV(model4, param_grid, cv=skf, scoring="roc_auc")
+grid_search.fit(
+    X_train, y_train_EC1
+)  # We can use y_train_EC1 to find general good params
+best_params = grid_search.best_params_
+
+# Update the GradientBoostingClassifier with the best parameters
+model4 = GradientBoostingClassifier(random_state=42, **best_params)
+
+# Combine models into a VotingClassifier with soft voting
+voting_clf_EC1 = VotingClassifier(
+    estimators=[("lgbm", model1_EC1), ("rf", model2), ("lr", model3), ("gbc", model4)],
+    voting="soft",
+)
+voting_clf_EC2 = VotingClassifier(
+    estimators=[("lgbm", model1_EC2), ("rf", model2), ("lr", model3), ("gbc", model4)],
+    voting="soft",
+)
+
+# Train and evaluate the ensemble model for EC1
+cv_scores_EC1 = cross_val_score(
+    voting_clf_EC1, X_train, y_train_EC1, cv=skf, scoring="roc_auc"
+)
+auc_EC1 = np.mean(cv_scores_EC1)
+
+# Train and evaluate the ensemble model for EC2
+cv_scores_EC2 = cross_val_score(
+    voting_clf_EC2, X_train, y_train_EC2, cv=skf, scoring="roc_auc"
+)
+auc_EC2 = np.mean(cv_scores_EC2)
+
+# Print the evaluation metric for each target
+print(f"Validation AUC for EC1: {auc_EC1}")
+print(f"Validation AUC for EC2: {auc_EC2}")
+print(f"Average Validation AUC: {(auc_EC1 + auc_EC2) / 2}")
+
+# Fit the ensemble models on the entire training set
+voting_clf_EC1.fit(X_train, y_train_EC1)
+voting_clf_EC2.fit(X_train, y_train_EC2)
+
+# Predict probabilities for the test set
+test_data["EC1"] = voting_clf_EC1.predict_proba(X_test)[:, 1]
+test_data["EC2"] = voting_clf_EC2.predict_proba(X_test)[:, 1]
+
+# Prepare the submission file
+submission = test_data[["id", "EC1", "EC2"]]
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e19.py

@@ -0,0 +1,83 @@
+import pandas as pd
+from catboost import CatBoostRegressor, Pool
+from sklearn.model_selection import KFold
+import numpy as np
+from sklearn.preprocessing import LabelEncoder
+
+
+def smape(y_true, y_pred):
+    denominator = (np.abs(y_true) + np.abs(y_pred)) / 2.0
+    diff = np.abs(y_true - y_pred) / denominator
+    diff[denominator == 0] = 0.0
+    return 100 * np.mean(diff)
+
+
+# Load data
+train = pd.read_csv("./input/train.csv")
+test = pd.read_csv("./input/test.csv")
+
+# Convert date to datetime and extract features
+for df in [train, test]:
+    df["date"] = pd.to_datetime(df["date"])
+    df["year"] = df["date"].dt.year
+    df["month"] = df["date"].dt.month
+    df["day"] = df["date"].dt.day
+    df["day_of_week"] = df["date"].dt.dayofweek  # Extract day of the week
+    # Create interaction terms
+    df["country_store"] = df["country"] + "_" + df["store"]
+    df["country_product"] = df["country"] + "_" + df["product"]
+    df["store_product"] = df["store"] + "_" + df["product"]
+
+# Encode new categorical features
+label_encoders = {}
+for col in ["country_store", "country_product", "store_product"]:
+    le = LabelEncoder()
+    train[col] = le.fit_transform(train[col])
+    test[col] = le.transform(test[col])
+    label_encoders[col] = le
+
+# Define categorical features including new interaction terms
+cat_features = [
+    "country",
+    "store",
+    "product",
+    "country_store",
+    "country_product",
+    "store_product",
+]
+
+# Prepare data for training
+X = train.drop(["num_sold", "date", "id"], axis=1)
+y = train["num_sold"]
+
+# Cross-validation
+kf = KFold(n_splits=10, shuffle=True, random_state=42)
+smape_scores = []
+
+for train_index, test_index in kf.split(X):
+    X_train, X_valid = X.iloc[train_index], X.iloc[test_index]
+    y_train, y_valid = y.iloc[train_index], y.iloc[test_index]
+
+    model = CatBoostRegressor(
+        iterations=2000,  # Increase iterations
+        learning_rate=0.05,  # Decrease learning rate
+        depth=8,  # Increase depth
+        loss_function="MAE",
+        cat_features=cat_features,
+        verbose=200,
+    )
+    model.fit(X_train, y_train, eval_set=(X_valid, y_valid), use_best_model=True)
+
+    preds = model.predict(X_valid)
+    score = smape(y_valid, preds)
+    smape_scores.append(score)
+
+print(f"Average SMAPE: {np.mean(smape_scores)}")
+
+# Prepare test data and make predictions
+test_data = test.drop(["date", "id"], axis=1)
+predictions = model.predict(test_data)
+
+# Save submission
+submission = pd.DataFrame({"id": test["id"], "num_sold": predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e20.py

@@ -0,0 +1,60 @@
+import pandas as pd
+import lightgbm as lgb
+from sklearn.metrics import mean_squared_error
+from sklearn.model_selection import train_test_split
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Preprocess the data
+train_data[["ID", "latitude", "longitude", "year", "week_no"]] = train_data[
+    "ID_LAT_LON_YEAR_WEEK"
+].str.split("_", expand=True)
+test_data[["ID", "latitude", "longitude", "year", "week_no"]] = test_data[
+    "ID_LAT_LON_YEAR_WEEK"
+].str.split("_", expand=True)
+
+# Convert to numeric types
+for col in ["latitude", "longitude", "year", "week_no"]:
+    train_data[col] = pd.to_numeric(train_data[col])
+    test_data[col] = pd.to_numeric(test_data[col])
+
+# One-hot encoding for 'week_no'
+train_data = pd.get_dummies(train_data, columns=["week_no"])
+test_data = pd.get_dummies(test_data, columns=["week_no"])
+
+# Align test_data columns with train_data
+test_data = test_data.reindex(columns=train_data.columns, fill_value=0)
+
+# Prepare the data for training
+X = train_data.drop(columns=["emission", "ID_LAT_LON_YEAR_WEEK", "ID"])
+y = train_data["emission"]
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Train the LightGBM model
+lgb_train = lgb.Dataset(X_train, y_train)
+lgb_eval = lgb.Dataset(X_val, y_val, reference=lgb_train)
+
+params = {"objective": "regression", "metric": "rmse", "verbose": -1}
+
+gbm = lgb.train(
+    params,
+    lgb_train,
+    num_boost_round=100,
+    valid_sets=lgb_eval,
+    early_stopping_rounds=10,
+)
+
+# Predict on validation set
+y_pred = gbm.predict(X_val, num_iteration=gbm.best_iteration)
+
+# Evaluate the model
+rmse = mean_squared_error(y_val, y_pred, squared=False)
+print(f"Validation RMSE: {rmse}")
+
+# Predict on test set and save submission
+test_features = test_data.drop(columns=["ID_LAT_LON_YEAR_WEEK", "ID", "emission"])
+test_data["emission"] = gbm.predict(test_features, num_iteration=gbm.best_iteration)
+submission = test_data[["ID_LAT_LON_YEAR_WEEK", "emission"]]
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e22.py

@@ -0,0 +1,64 @@
+import pandas as pd
+import lightgbm as lgb
+from sklearn.model_selection import StratifiedKFold
+from sklearn.metrics import f1_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Identify common categorical columns
+categorical_features = train_data.select_dtypes(include=["object"]).columns
+common_categorical_features = [
+    feature for feature in categorical_features if feature in test_data.columns
+]
+
+# Convert common categorical columns to 'category' data type
+for feature in common_categorical_features:
+    train_data[feature] = train_data[feature].astype("category")
+    test_data[feature] = test_data[feature].astype("category")
+
+# Separate features and target
+X = train_data.drop(["id", "outcome"], axis=1)
+y = train_data["outcome"]
+X_test = test_data.drop(["id"], axis=1)
+
+# Prepare for cross-validation
+folds = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)
+predictions = pd.DataFrame()
+scores = []
+
+# Perform 10-fold cross-validation
+for fold_n, (train_index, valid_index) in enumerate(folds.split(X, y)):
+    X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]
+    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
+
+    # Train the model
+    model = lgb.LGBMClassifier(objective="multiclass", random_state=42)
+    model.fit(
+        X_train,
+        y_train,
+        categorical_feature=common_categorical_features,
+        eval_set=[(X_valid, y_valid)],
+        early_stopping_rounds=50,
+        verbose=False,
+    )
+
+    # Make predictions
+    y_pred = model.predict(X_valid)
+
+    # Calculate the F1 score
+    score = f1_score(y_valid, y_pred, average="micro")
+    scores.append(score)
+
+    # Predict on test set
+    predictions[f"fold_{fold_n}"] = model.predict(X_test)
+
+# Print the average F1 score across all folds
+print(f"Average F1-Score: {sum(scores) / len(scores)}")
+
+# Prepare submission file
+submission = pd.DataFrame()
+submission["id"] = test_data["id"]
+submission["outcome"] = predictions.mode(axis=1)[0]
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e23.py

@@ -0,0 +1,70 @@
+import pandas as pd
+import numpy as np
+from sklearn.model_selection import KFold
+from sklearn.metrics import roc_auc_score
+from sklearn.feature_selection import RFECV
+import lightgbm as lgb
+from bayes_opt import BayesianOptimization
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "defects"], axis=1)
+y = train_data["defects"]
+X_test = test_data.drop("id", axis=1)
+test_ids = test_data["id"]
+
+# Initialize LightGBM model with the best parameters from previous optimization
+best_params = {
+    "num_leaves": 31,
+    "learning_rate": 0.05,
+    "subsample": 0.8,
+    "colsample_bytree": 0.8,
+    "max_depth": 15,
+    "reg_alpha": 0.5,
+    "reg_lambda": 0.5,
+    "objective": "binary",
+    "metric": "auc",
+    "verbosity": -1,
+    "n_jobs": -1,
+    "random_state": 42,
+}
+lgb_model = lgb.LGBMClassifier(**best_params)
+
+# Perform feature selection using RFECV
+rfecv = RFECV(estimator=lgb_model, step=1, cv=KFold(10), scoring="roc_auc", n_jobs=-1)
+rfecv.fit(X, y)
+
+# Print the optimal number of features
+print(f"Optimal number of features: {rfecv.n_features_}")
+
+# Select the optimal features
+X_selected = rfecv.transform(X)
+X_test_selected = rfecv.transform(X_test)
+
+# Retrain the model with the selected features
+lgb_model.fit(X_selected, y)
+
+# Predict on the test set with the selected features
+final_predictions = lgb_model.predict_proba(X_test_selected)[:, 1]
+
+# Save the submission file
+submission = pd.DataFrame({"id": test_ids, "defects": final_predictions})
+submission.to_csv("./working/submission.csv", index=False)
+
+# Evaluate the model with selected features using cross-validation
+auc_scores = []
+kf = KFold(n_splits=10, shuffle=True, random_state=42)
+for train_index, valid_index in kf.split(X_selected):
+    X_train, X_valid = X_selected[train_index], X_selected[valid_index]
+    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]
+    lgb_model.fit(X_train, y_train)
+    y_pred = lgb_model.predict_proba(X_valid)[:, 1]
+    auc_score = roc_auc_score(y_valid, y_pred)
+    auc_scores.append(auc_score)
+
+# Print the mean AUC score
+mean_auc_score = np.mean(auc_scores)
+print(f"Mean AUC Score with Selected Features: {mean_auc_score}")

examples/playground-series-s3e24.py

@@ -0,0 +1,103 @@
+import pandas as pd
+import lightgbm as lgb
+from sklearn.model_selection import train_test_split, KFold
+from sklearn.metrics import roc_auc_score
+from sklearn.preprocessing import StandardScaler
+from bayes_opt import BayesianOptimization
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "smoking"], axis=1)
+y = train_data["smoking"]
+X_test = test_data.drop("id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Scale the features
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train)
+X_val_scaled = scaler.transform(X_val)
+X_test_scaled = scaler.transform(X_test)
+
+
+# Define the LightGBM cross-validation function
+def lgb_cv(
+    learning_rate,
+    num_leaves,
+    min_child_samples,
+    subsample,
+    colsample_bytree,
+    max_depth,
+    reg_alpha,
+    reg_lambda,
+    n_estimators,
+):
+    params = {
+        "objective": "binary",
+        "metric": "auc",
+        "boosting_type": "gbdt",
+        "learning_rate": max(min(learning_rate, 1), 0),
+        "n_estimators": int(n_estimators),
+        "verbose": -1,
+        "num_leaves": int(num_leaves),
+        "min_child_samples": int(min_child_samples),
+        "subsample": max(min(subsample, 1), 0),
+        "colsample_bytree": max(min(colsample_bytree, 1), 0),
+        "max_depth": int(max_depth),
+        "reg_alpha": max(reg_alpha, 0),
+        "reg_lambda": max(reg_lambda, 0),
+    }
+    cv_result = lgb.cv(
+        params,
+        lgb.Dataset(X_train_scaled, label=y_train),
+        nfold=10,
+        seed=42,
+        stratified=True,
+        verbose_eval=200,
+        metrics=["auc"],
+    )
+    return max(cv_result["auc-mean"])
+
+
+# Define the parameter bounds
+param_bounds = {
+    "learning_rate": (0.01, 0.2),
+    "num_leaves": (20, 60),
+    "min_child_samples": (5, 50),
+    "subsample": (0.6, 1.0),
+    "colsample_bytree": (0.6, 1.0),
+    "max_depth": (5, 15),
+    "reg_alpha": (0, 1),
+    "reg_lambda": (0, 1),
+    "n_estimators": (100, 1000),  # Increased range for n_estimators
+}
+
+# Perform Bayesian optimization with increased initial points and iterations
+optimizer = BayesianOptimization(f=lgb_cv, pbounds=param_bounds, random_state=42)
+optimizer.maximize(init_points=10, n_iter=50)
+
+# Retrieve the best parameters
+best_params = optimizer.max["params"]
+best_params["num_leaves"] = int(best_params["num_leaves"])
+best_params["min_child_samples"] = int(best_params["min_child_samples"])
+best_params["max_depth"] = int(best_params["max_depth"])
+best_params["n_estimators"] = int(best_params["n_estimators"])
+
+# Train and validate the model with the best parameters
+final_gbm = lgb.LGBMClassifier(**best_params)
+final_gbm.fit(X_train_scaled, y_train)
+val_predictions = final_gbm.predict_proba(X_val_scaled)[:, 1]
+val_auc = roc_auc_score(y_val, val_predictions)
+print(f"Validation AUC score: {val_auc}")
+
+# Train the model on the full dataset with the best parameters and make predictions on the scaled test set
+final_gbm.fit(scaler.fit_transform(X), y)
+predictions = final_gbm.predict_proba(X_test_scaled)[:, 1]
+
+# Prepare the submission file
+submission = pd.DataFrame({"id": test_data["id"], "smoking": predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e25.py

@@ -0,0 +1,63 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split, GridSearchCV
+from sklearn.svm import SVR
+from sklearn.metrics import median_absolute_error
+from sklearn.preprocessing import StandardScaler, PolynomialFeatures
+
+# Load the dataset
+train_data = pd.read_csv("./input/train.csv")
+
+# Separate features and target
+X = train_data.drop(["id", "Hardness"], axis=1)
+y = train_data["Hardness"]
+
+# Splitting the dataset into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Create interaction terms
+poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
+X_train_poly = poly.fit_transform(X_train)
+X_val_poly = poly.transform(X_val)
+
+# Standardize features
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train_poly)
+X_val_scaled = scaler.transform(X_val_poly)
+
+# Initialize the SVR model
+svr = SVR(kernel="rbf")
+
+# Define the expanded parameter grid
+param_grid = {
+    "C": [0.1, 0.5, 1, 1.5, 2, 2.5, 3],
+    "gamma": [0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3],
+    "epsilon": [0.001, 0.005, 0.01, 0.015, 0.02, 0.025, 0.03],
+}
+
+# Initialize GridSearchCV
+grid_search = GridSearchCV(
+    svr, param_grid, cv=5, scoring="neg_median_absolute_error", verbose=1, n_jobs=-1
+)
+
+# Fit the model
+grid_search.fit(X_train_scaled, y_train)
+
+# Best parameters
+print(f"Best parameters: {grid_search.best_params_}")
+
+# Predict on the validation set using the best estimator
+best_model = grid_search.best_estimator_
+predictions = best_model.predict(X_val_scaled)
+
+# Evaluate the model
+medae = median_absolute_error(y_val, predictions)
+print(f"Median Absolute Error: {medae}")
+
+# Prepare submission
+test_data = pd.read_csv("./input/test.csv")
+X_test = test_data.drop(["id"], axis=1)
+X_test_poly = poly.transform(X_test)
+X_test_scaled = scaler.transform(X_test_poly)
+test_predictions = best_model.predict(X_test_scaled)
+submission = pd.DataFrame({"id": test_data["id"], "Hardness": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e26.py

@@ -0,0 +1,65 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import log_loss
+from sklearn.preprocessing import OneHotEncoder, StandardScaler
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["Status", "id"], axis=1)
+y = train_data["Status"]
+X_test = test_data.drop("id", axis=1)
+
+# Preprocessing for numerical data
+numerical_transformer = StandardScaler()
+
+# Preprocessing for categorical data
+categorical_transformer = OneHotEncoder(handle_unknown="ignore")
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, X.select_dtypes(exclude=["object"]).columns),
+        ("cat", categorical_transformer, X.select_dtypes(include=["object"]).columns),
+    ]
+)
+
+# Define the model
+model = RandomForestClassifier(n_estimators=100, random_state=0)
+
+# Bundle preprocessing and modeling code in a pipeline
+clf = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Split data into train and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Preprocessing of training data, fit model
+clf.fit(X_train, y_train)
+
+# Preprocessing of validation data, get predictions
+preds = clf.predict_proba(X_valid)
+
+# Evaluate the model
+score = log_loss(pd.get_dummies(y_valid), preds)
+print("Log Loss:", score)
+
+# Preprocessing of test data, fit model
+test_preds = clf.predict_proba(X_test)
+
+# Generate submission file
+output = pd.DataFrame(
+    {
+        "id": test_data.id,
+        "Status_C": test_preds[:, 0],
+        "Status_CL": test_preds[:, 1],
+        "Status_D": test_preds[:, 2],
+    }
+)
+output.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e3.py

@@ -0,0 +1,65 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import roc_auc_score
+from sklearn.preprocessing import OneHotEncoder, StandardScaler
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["Attrition", "id"], axis=1)
+y = train_data["Attrition"]
+X_test = test_data.drop("id", axis=1)
+
+# Identify categorical and numerical columns
+categorical_cols = X.select_dtypes(include=["object"]).columns
+numerical_cols = X.select_dtypes(include=["int64", "float64"]).columns
+
+# Create the preprocessing pipelines for both numeric and categorical data
+numeric_transformer = Pipeline(steps=[("scaler", StandardScaler())])
+
+categorical_transformer = Pipeline(
+    steps=[("onehot", OneHotEncoder(handle_unknown="ignore"))]
+)
+
+# Combine preprocessing steps
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numeric_transformer, numerical_cols),
+        ("cat", categorical_transformer, categorical_cols),
+    ]
+)
+
+# Create a pipeline that combines the preprocessor with a classifier
+model = Pipeline(
+    steps=[
+        ("preprocessor", preprocessor),
+        ("classifier", LogisticRegression(solver="liblinear")),
+    ]
+)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Train the model
+model.fit(X_train, y_train)
+
+# Predict probabilities on the validation set
+y_pred_proba = model.predict_proba(X_val)[:, 1]
+
+# Calculate the AUC
+auc = roc_auc_score(y_val, y_pred_proba)
+print(f"Validation AUC: {auc}")
+
+# Predict probabilities on the test set
+test_pred_proba = model.predict_proba(X_test)[:, 1]
+
+# Create a submission file
+submission = pd.DataFrame(
+    {"EmployeeNumber": test_data["id"], "Attrition": test_pred_proba}
+)
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e5.py

@@ -0,0 +1,39 @@
+import pandas as pd
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import cohen_kappa_score
+import lightgbm as lgb
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["Id", "quality"], axis=1)
+y = train_data["quality"]
+X_test = test_data.drop("Id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Train the model
+model = lgb.LGBMRegressor(random_state=42)
+model.fit(X_train, y_train)
+
+# Make predictions
+val_predictions = model.predict(X_val)
+val_predictions = np.round(val_predictions).astype(int)  # Round to the nearest integer
+
+# Evaluate the model
+kappa_score = cohen_kappa_score(y_val, val_predictions, weights="quadratic")
+print(f"Quadratic Weighted Kappa score on validation set: {kappa_score}")
+
+# Make predictions on the test set
+test_predictions = model.predict(X_test)
+test_predictions = np.round(test_predictions).astype(
+    int
+)  # Round to the nearest integer
+
+# Prepare the submission file
+submission = pd.DataFrame({"Id": test_data["Id"], "quality": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e7.py

@@ -0,0 +1,33 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "booking_status"], axis=1)
+y = train_data["booking_status"]
+X_test = test_data.drop("id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize and train the logistic regression model
+model = LogisticRegression(max_iter=1000, random_state=42)
+model.fit(X_train, y_train)
+
+# Predict on the validation set
+val_predictions = model.predict_proba(X_val)[:, 1]
+val_roc_auc = roc_auc_score(y_val, val_predictions)
+print(f"Validation ROC AUC: {val_roc_auc}")
+
+# Train the model on the full training data and predict on the test set
+model.fit(X, y)
+test_predictions = model.predict_proba(X_test)[:, 1]
+
+# Save the predictions in the submission format
+submission = pd.DataFrame({"id": test_data["id"], "booking_status": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s3e9.py

@@ -0,0 +1,35 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.metrics import mean_squared_error
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Prepare the data
+X = train_data.drop(["id", "Strength"], axis=1)
+y = train_data["Strength"]
+X_test = test_data.drop("id", axis=1)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize and train the model
+model = GradientBoostingRegressor(random_state=42)
+model.fit(X_train, y_train)
+
+# Predict on validation set
+y_pred_val = model.predict(X_val)
+
+# Evaluate the model
+rmse = np.sqrt(mean_squared_error(y_val, y_pred_val))
+print(f"Validation RMSE: {rmse}")
+
+# Predict on test set
+test_predictions = model.predict(X_test)
+
+# Save the predictions to a CSV file
+submission = pd.DataFrame({"id": test_data["id"], "Strength": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/playground-series-s4e1.py

@@ -0,0 +1,58 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import OneHotEncoder, StandardScaler
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.metrics import roc_auc_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["Exited", "id", "CustomerId", "Surname"], axis=1)
+y = train_data["Exited"]
+X_test = test_data.drop(["id", "CustomerId", "Surname"], axis=1)
+
+# Preprocessing for numerical data
+numerical_transformer = StandardScaler()
+
+# Preprocessing for categorical data
+categorical_transformer = OneHotEncoder(handle_unknown="ignore")
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, X.select_dtypes(exclude=["object"]).columns),
+        ("cat", categorical_transformer, X.select_dtypes(include=["object"]).columns),
+    ]
+)
+
+# Define the model
+model = GradientBoostingClassifier()
+
+# Bundle preprocessing and modeling code in a pipeline
+clf = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Split data into train and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, test_size=0.2, random_state=0
+)
+
+# Preprocessing of training data, fit model
+clf.fit(X_train, y_train)
+
+# Preprocessing of validation data, get predictions
+preds = clf.predict_proba(X_valid)[:, 1]
+
+# Evaluate the model
+score = roc_auc_score(y_valid, preds)
+print(f"ROC AUC score: {score}")
+
+# Preprocessing of test data, fit model
+preds_test = clf.predict_proba(X_test)[:, 1]
+
+# Save test predictions to file
+output = pd.DataFrame({"id": test_data.id, "Exited": preds_test})
+output.to_csv("./working/submission.csv", index=False)

examples/playground-series-s4e2.py

@@ -0,0 +1,65 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.metrics import accuracy_score
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["NObeyesdad", "id"], axis=1)
+y = train_data["NObeyesdad"]
+X_test = test_data.drop("id", axis=1)
+
+# Identify categorical and numerical columns
+categorical_cols = [cname for cname in X.columns if X[cname].dtype == "object"]
+numerical_cols = [
+    cname for cname in X.columns if X[cname].dtype in ["int64", "float64"]
+]
+
+# Preprocessing for numerical data
+numerical_transformer = StandardScaler()
+
+# Preprocessing for categorical data
+categorical_transformer = OneHotEncoder(handle_unknown="ignore")
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, numerical_cols),
+        ("cat", categorical_transformer, categorical_cols),
+    ]
+)
+
+# Define the model
+model = RandomForestClassifier(n_estimators=100, random_state=0)
+
+# Bundle preprocessing and modeling code in a pipeline
+clf = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Split data into train and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Preprocessing of training data, fit model
+clf.fit(X_train, y_train)
+
+# Preprocessing of validation data, get predictions
+preds = clf.predict(X_valid)
+
+# Evaluate the model
+score = accuracy_score(y_valid, preds)
+print("Accuracy:", score)
+
+# Preprocessing of test data, fit model
+preds_test = clf.predict(X_test)
+
+# Save test predictions to file
+output = pd.DataFrame({"id": test_data.id, "NObeyesdad": preds_test})
+output.to_csv("./working/submission.csv", index=False)

examples/santa-2019-revenge-of-the-accountants.py

@@ -0,0 +1,112 @@
+import pandas as pd
+import numpy as np
+from math import exp
+
+# Load the data
+family_data = pd.read_csv("./input/family_data.csv")
+sample_submission = pd.read_csv("./input/sample_submission.csv")
+
+# Constants
+N_DAYS = 100
+N_FAMILY = len(family_data)
+MAX_OCCUPANCY = 300
+MIN_OCCUPANCY = 125
+
+
+# Cost function components
+def preference_cost_matrix(family_data):
+    cost_matrix = np.zeros((N_FAMILY, N_DAYS + 1), dtype=np.int64)
+    for i in range(N_FAMILY):
+        family = family_data.iloc[i]
+        n_people = family["n_people"]
+        for j in range(10):
+            day = family[f"choice_{j}"]
+            if j == 0:
+                cost_matrix[i, day] = 0
+            elif j == 1:
+                cost_matrix[i, day] = 50
+            elif j == 2:
+                cost_matrix[i, day] = 50 + 9 * n_people
+            elif j == 3:
+                cost_matrix[i, day] = 100 + 9 * n_people
+            elif j == 4:
+                cost_matrix[i, day] = 200 + 9 * n_people
+            elif j == 5:
+                cost_matrix[i, day] = 200 + 18 * n_people
+            elif j == 6:
+                cost_matrix[i, day] = 300 + 18 * n_people
+            elif j == 7:
+                cost_matrix[i, day] = 300 + 36 * n_people
+            elif j == 8:
+                cost_matrix[i, day] = 400 + 36 * n_people
+            elif j == 9:
+                cost_matrix[i, day] = 500 + 36 * n_people + 199 * n_people
+        cost_matrix[i, 0] = 500 + 36 * n_people + 398 * n_people
+    return cost_matrix
+
+
+def accounting_penalty(occupancy):
+    penalties = np.zeros(N_DAYS + 1)
+    for day in range(N_DAYS - 1, -1, -1):
+        Nd = occupancy[day]
+        Nd_next = occupancy[day + 1]
+        penalties[day] = max(0, (Nd - 125) / 400 * Nd ** (0.5 + abs(Nd - Nd_next) / 50))
+    return penalties.sum()
+
+
+# Simulated annealing
+def simulated_annealing(family_data, sample_submission, cost_matrix):
+    best = sample_submission["assigned_day"].values
+    occupancy = np.zeros(N_DAYS + 1, dtype=int)
+    for i, day in enumerate(best):
+        occupancy[day] += family_data.iloc[i]["n_people"]
+    occupancy[0] = occupancy[N_DAYS]  # Occupancy for the "zeroth" day
+    best_score = cost_matrix[np.arange(N_FAMILY), best].sum() + accounting_penalty(
+        occupancy
+    )
+    temperature = 1.0
+    alpha = 0.99
+    for step in range(10000):
+        # Create new candidate solution
+        family_id = np.random.choice(range(N_FAMILY))
+        old_day = best[family_id]
+        new_day = np.random.choice(range(1, N_DAYS + 1))
+        best[family_id] = new_day
+
+        # Calculate the cost
+        new_occupancy = occupancy.copy()
+        new_occupancy[old_day] -= family_data.iloc[family_id]["n_people"]
+        new_occupancy[new_day] += family_data.iloc[family_id]["n_people"]
+        new_occupancy[0] = new_occupancy[N_DAYS]  # Occupancy for the "zeroth" day
+        if any((new_occupancy < MIN_OCCUPANCY) | (new_occupancy > MAX_OCCUPANCY)):
+            best[family_id] = old_day  # Revert changes
+            continue
+        new_score = cost_matrix[np.arange(N_FAMILY), best].sum() + accounting_penalty(
+            new_occupancy
+        )
+
+        # Acceptance probability
+        if new_score < best_score or np.random.rand() < exp(
+            -(new_score - best_score) / temperature
+        ):
+            best_score = new_score
+            occupancy = new_occupancy
+        else:
+            best[family_id] = old_day  # Revert changes
+
+        # Cool down
+        temperature *= alpha
+
+    return best, best_score
+
+
+# Run the optimization
+cost_matrix = preference_cost_matrix(family_data)
+best_schedule, best_score = simulated_annealing(
+    family_data, sample_submission, cost_matrix
+)
+
+# Output the result
+print(f"Best score: {best_score}")
+sample_submission["assigned_day"] = best_schedule
+sample_submission.to_csv("./working/submission.csv", index=False)

examples/scrabble-player-rating.py

@@ -0,0 +1,54 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error
+from math import sqrt
+
+# Load the data
+games = pd.read_csv("./input/games.csv")
+turns = pd.read_csv("./input/turns.csv")
+train = pd.read_csv("./input/train.csv")
+
+# Merge the datasets on game_id
+merged_data = pd.merge(train, games, on="game_id")
+merged_data = pd.merge(
+    merged_data,
+    turns.groupby("game_id").agg({"points": "sum"}).reset_index(),
+    on="game_id",
+)
+
+# Prepare the features and target variable
+X = merged_data[["game_duration_seconds", "winner", "points"]]
+y = merged_data["rating"]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize the model
+model = RandomForestRegressor(n_estimators=100, random_state=42)
+
+# Train the model
+model.fit(X_train, y_train)
+
+# Predict on the validation set
+y_pred = model.predict(X_val)
+
+# Calculate the RMSE
+rmse = sqrt(mean_squared_error(y_val, y_pred))
+print(f"Validation RMSE: {rmse}")
+
+# Prepare the test set
+test = pd.read_csv("./input/test.csv")
+test_merged = pd.merge(test, games, on="game_id")
+test_merged = pd.merge(
+    test_merged,
+    turns.groupby("game_id").agg({"points": "sum"}).reset_index(),
+    on="game_id",
+)
+X_test = test_merged[["game_duration_seconds", "winner", "points"]]
+
+# Predict on the test set
+test["rating"] = model.predict(X_test)
+
+# Save the predictions to a CSV file
+test[["game_id", "rating"]].to_csv("./working/submission.csv", index=False)

examples/sentiment-analysis-on-movie-reviews.py

@@ -0,0 +1,55 @@
+import pandas as pd
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score
+
+# Load the training data
+train_data = pd.read_csv("./input/train.tsv", sep="\t")
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(
+    train_data["Phrase"], train_data["Sentiment"], test_size=0.2, random_state=42
+)
+
+# Initialize a TF-IDF Vectorizer
+tfidf_vectorizer = TfidfVectorizer(stop_words="english", ngram_range=(1, 2))
+
+# Fit and transform the training data
+X_train_tfidf = tfidf_vectorizer.fit_transform(X_train.astype(str))
+
+# Transform the validation data
+X_val_tfidf = tfidf_vectorizer.transform(X_val.astype(str))
+
+# Initialize the Logistic Regression model
+logistic_regression_model = LogisticRegression(random_state=42)
+
+# Train the model
+logistic_regression_model.fit(X_train_tfidf, y_train)
+
+# Predict the sentiments on the validation set
+y_val_pred = logistic_regression_model.predict(X_val_tfidf)
+
+# Calculate the accuracy on the validation set
+accuracy = accuracy_score(y_val, y_val_pred)
+print(f"Validation Accuracy: {accuracy}")
+
+# Load the test data
+test_data = pd.read_csv("./input/test.tsv", sep="\t")
+
+# Preprocess the test data by filling NaN values with an empty string
+test_data["Phrase"] = test_data["Phrase"].fillna("")
+
+# Transform the test data using the same vectorizer
+X_test_tfidf = tfidf_vectorizer.transform(test_data["Phrase"].astype(str))
+
+# Predict the sentiments on the test set
+test_predictions = logistic_regression_model.predict(X_test_tfidf)
+
+# Prepare the submission file
+submission = pd.DataFrame(
+    {"PhraseId": test_data["PhraseId"], "Sentiment": test_predictions}
+)
+
+# Save the submission file
+submission.to_csv("./working/submission.csv", index=False)

examples/spaceship-titanic.py

@@ -0,0 +1,78 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.metrics import accuracy_score
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate target from predictors
+y = train_data["Transported"]
+X = train_data.drop(["Transported"], axis=1)
+
+# Select categorical columns with relatively low cardinality
+categorical_cols = [
+    cname
+    for cname in X.columns
+    if X[cname].nunique() < 10 and X[cname].dtype == "object"
+]
+
+# Select numerical columns
+numerical_cols = [
+    cname for cname in X.columns if X[cname].dtype in ["int64", "float64"]
+]
+
+# Preprocessing for numerical data
+numerical_transformer = SimpleImputer(strategy="median")
+
+# Preprocessing for categorical data
+categorical_transformer = Pipeline(
+    steps=[
+        ("imputer", SimpleImputer(strategy="most_frequent")),
+        ("onehot", OneHotEncoder(handle_unknown="ignore")),
+    ]
+)
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, numerical_cols),
+        ("cat", categorical_transformer, categorical_cols),
+    ]
+)
+
+# Define the model
+model = RandomForestClassifier(n_estimators=100, random_state=0)
+
+# Bundle preprocessing and modeling code in a pipeline
+clf = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Split data into train and validation sets
+X_train, X_valid, y_train, y_valid = train_test_split(
+    X, y, train_size=0.8, test_size=0.2, random_state=0
+)
+
+# Preprocessing of training data, fit model
+clf.fit(X_train, y_train)
+
+# Preprocessing of validation data, get predictions
+preds = clf.predict(X_valid)
+
+# Evaluate the model
+score = accuracy_score(y_valid, preds)
+print("Accuracy:", score)
+
+# Preprocessing of test data, fit model
+preprocessed_test_data = clf.named_steps["preprocessor"].transform(test_data)
+
+# Get test predictions
+test_preds = clf.named_steps["model"].predict(preprocessed_test_data)
+
+# Save test predictions to file
+output = pd.DataFrame({"PassengerId": test_data.PassengerId, "Transported": test_preds})
+output.to_csv("./working/submission.csv", index=False)

examples/tabular-playground-series-apr-2021.py

@@ -0,0 +1,54 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import cross_val_score
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Features and target
+X = train_data.drop(["Survived", "PassengerId", "Name", "Ticket", "Cabin"], axis=1)
+y = train_data["Survived"]
+
+# Preprocessing for numerical data
+numerical_transformer = SimpleImputer(strategy="median")
+
+# Preprocessing for categorical data
+categorical_cols = [cname for cname in X.columns if X[cname].dtype == "object"]
+categorical_transformer = Pipeline(
+    steps=[
+        ("imputer", SimpleImputer(strategy="most_frequent")),
+        ("onehot", OneHotEncoder(handle_unknown="ignore")),
+    ]
+)
+
+# Bundle preprocessing for numerical and categorical data
+preprocessor = ColumnTransformer(
+    transformers=[
+        ("num", numerical_transformer, ["Age", "Fare"]),
+        ("cat", categorical_transformer, categorical_cols),
+    ]
+)
+
+# Define the model
+model = RandomForestClassifier(n_estimators=100, random_state=0)
+
+# Bundle preprocessing and modeling code in a pipeline
+clf = Pipeline(steps=[("preprocessor", preprocessor), ("model", model)])
+
+# Cross-validation scores
+scores = cross_val_score(clf, X, y, cv=10, scoring="accuracy")
+print(f"Average cross-validation score: {scores.mean():.4f}")
+
+# Preprocessing of test data, fit model
+clf.fit(X, y)
+test_X = test_data.drop(["PassengerId", "Name", "Ticket", "Cabin"], axis=1)
+test_preds = clf.predict(test_X)
+
+# Save test predictions to file
+output = pd.DataFrame({"PassengerId": test_data.PassengerId, "Survived": test_preds})
+output.to_csv("./working/submission.csv", index=False)

examples/tabular-playground-series-apr-2022.py

@@ -0,0 +1,43 @@
+import pandas as pd
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.metrics import roc_auc_score
+from sklearn.model_selection import train_test_split
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+train_labels = pd.read_csv("./input/train_labels.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Aggregate features for each sequence
+agg_funcs = ["mean", "std", "min", "max"]
+train_features = train_data.groupby("sequence").agg(agg_funcs)
+test_features = test_data.groupby("sequence").agg(agg_funcs)
+
+# Flatten multi-level columns
+train_features.columns = [
+    "_".join(col).strip() for col in train_features.columns.values
+]
+test_features.columns = ["_".join(col).strip() for col in test_features.columns.values]
+
+# Split the data into train and validation sets
+X_train, X_val, y_train, y_val = train_test_split(
+    train_features, train_labels["state"], test_size=0.2, random_state=42
+)
+
+# Initialize and train the Random Forest classifier
+rf = RandomForestClassifier(n_estimators=100, random_state=42)
+rf.fit(X_train, y_train)
+
+# Predict probabilities for the validation set
+val_probs = rf.predict_proba(X_val)[:, 1]
+
+# Calculate the AUC-ROC score
+auc_score = roc_auc_score(y_val, val_probs)
+print(f"AUC-ROC score: {auc_score}")
+
+# Predict probabilities for the test set
+test_probs = rf.predict_proba(test_features)[:, 1]
+
+# Create the submission file
+submission = pd.DataFrame({"sequence": test_features.index, "state": test_probs})
+submission.to_csv("./working/submission.csv", index=False)

examples/tabular-playground-series-aug-2021.py

@@ -0,0 +1,46 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.preprocessing import StandardScaler
+from sklearn.metrics import mean_squared_error
+import numpy as np
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(["id", "loss"], axis=1)
+y = train_data["loss"]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Scale the features
+scaler = StandardScaler()
+X_train_scaled = scaler.fit_transform(X_train)
+X_val_scaled = scaler.transform(X_val)
+
+# Initialize the model
+model = GradientBoostingRegressor(random_state=42)
+
+# Fit the model
+model.fit(X_train_scaled, y_train)
+
+# Predict on the validation set
+y_pred = model.predict(X_val_scaled)
+
+# Calculate the RMSE
+rmse = np.sqrt(mean_squared_error(y_val, y_pred))
+print(f"Validation RMSE: {rmse}")
+
+# Prepare the test set
+X_test = test_data.drop("id", axis=1)
+X_test_scaled = scaler.transform(X_test)
+
+# Predict on the test set
+test_predictions = model.predict(X_test_scaled)
+
+# Create the submission file
+submission = pd.DataFrame({"id": test_data["id"], "loss": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)

examples/tabular-playground-series-aug-2022.py

@@ -0,0 +1,69 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import roc_auc_score
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.impute import SimpleImputer
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Preprocess the data
+features = train_data.columns.drop(["id", "failure"])
+X = train_data[features]
+y = train_data["failure"]
+X_test = test_data[features]
+
+# Fill missing values with median for numerical columns
+num_cols = X.select_dtypes(exclude="object").columns
+imputer = SimpleImputer(strategy="median")
+X[num_cols] = imputer.fit_transform(X[num_cols])
+X_test[num_cols] = imputer.transform(X_test[num_cols])
+
+# One-hot encode categorical features
+cat_cols = X.select_dtypes(include="object").columns
+encoder = OneHotEncoder(handle_unknown="ignore", sparse=False)
+X_encoded = pd.DataFrame(
+    encoder.fit_transform(X[cat_cols]), columns=encoder.get_feature_names_out(cat_cols)
+)
+X_test_encoded = pd.DataFrame(
+    encoder.transform(X_test[cat_cols]), columns=encoder.get_feature_names_out(cat_cols)
+)
+
+# One-hot encoding removed index; put it back
+X_encoded.index = X.index
+X_test_encoded.index = X_test.index
+
+# Remove categorical columns (will replace with one-hot encoding)
+num_X = X.drop(cat_cols, axis=1)
+num_X_test = X_test.drop(cat_cols, axis=1)
+
+# Add one-hot encoded columns to numerical features
+X_preprocessed = pd.concat([num_X, X_encoded], axis=1)
+X_test_preprocessed = pd.concat([num_X_test, X_test_encoded], axis=1)
+
+# Convert all feature names to strings to avoid TypeError
+X_preprocessed.columns = X_preprocessed.columns.astype(str)
+X_test_preprocessed.columns = X_test_preprocessed.columns.astype(str)
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(
+    X_preprocessed, y, test_size=0.2, random_state=0
+)
+
+# Train the Logistic Regression model
+model = LogisticRegression(max_iter=1000)
+model.fit(X_train, y_train)
+
+# Evaluate the model
+val_predictions = model.predict_proba(X_val)[:, 1]
+val_auc = roc_auc_score(y_val, val_predictions)
+print(f"Validation ROC AUC Score: {val_auc}")
+
+# Predict on test data
+test_predictions = model.predict_proba(X_test_preprocessed)[:, 1]
+
+# Save the predictions to a CSV file
+output = pd.DataFrame({"id": test_data.id, "failure": test_predictions})
+output.to_csv("./working/submission.csv", index=False)

examples/tabular-playground-series-dec-2021.py

@@ -0,0 +1,31 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.metrics import accuracy_score
+
+# Load the data
+train_data = pd.read_csv("./input/train.csv")
+test_data = pd.read_csv("./input/test.csv")
+
+# Separate features and target
+X = train_data.drop(columns=["Id", "Cover_Type"])
+y = train_data["Cover_Type"]
+
+# Split the data into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
+
+# Initialize and train the Random Forest Classifier
+model = RandomForestClassifier(random_state=42)
+model.fit(X_train, y_train)
+
+# Predict on the validation set and calculate accuracy
+val_predictions = model.predict(X_val)
+accuracy = accuracy_score(y_val, val_predictions)
+print(f"Validation Accuracy: {accuracy}")
+
+# Predict on the test set
+test_predictions = model.predict(test_data.drop(columns=["Id"]))
+
+# Save the predictions to a CSV file
+submission = pd.DataFrame({"Id": test_data["Id"], "Cover_Type": test_predictions})
+submission.to_csv("./working/submission.csv", index=False)