{"id":28356,"date":"2022-01-21T12:10:56","date_gmt":"2022-01-21T06:40:56","guid":{"rendered":"https:\/\/www.subex.com\/?p=28356"},"modified":"2022-01-21T12:10:56","modified_gmt":"2022-01-21T06:40:56","slug":"an-introduction-to-efficient-hypermeter-optimization-for-xgboost-model-using-optuna","status":"publish","type":"post","link":"https:\/\/dev.enki.studio\/test\/?p=28356","title":{"rendered":"An Introduction to efficient Hypermeter optimization for XGBoost model using Optuna"},"content":{"rendered":"<p><strong><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28360 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-1-1.jpg\" alt=\"Efficient Hyperparam\" width=\"1050\" height=\"656\" \/><\/strong><\/p>\n<p><strong>Introduction :<\/strong><\/p>\n<p>Hyperparameter optimization is the science of tuning or choosing the best set of hyperparameters for a learning algorithm. A set of optimal hyperparameter has a big impact on the performance of any machine learning algorithm. It is one of the most time-consuming yet a crucial step in machine learning training pipeline.<\/p>\n<p>A Machine learning model has two types of tunable parameter :<\/p>\n<ul>\n<li>Model parameters<\/li>\n<li>Model hyperparameters<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28362 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-3-1.jpg\" alt=\"Efficient Hyperparam\" width=\"353\" height=\"242\" \/><\/p>\n<p>Model parameters vs Model hyperparameters (<a href=\"https:\/\/www.slideshare.net\/AliceZheng3\/evaluating-machine-learning-models-a-beginners-guide\">source<\/a>)<\/p>\n<p><strong><em>Model parameters<\/em><\/strong>\u00a0are learned during the training phase of a model or classifier. For example :<\/p>\n<ul>\n<li>coefficients in logistic regression or liner regression<\/li>\n<li>weights in an artificial neural network<\/li>\n<\/ul>\n<p><strong><em>Model Hyperparameters\u00a0<\/em><\/strong>are set by user before the model training phase. For example :<\/p>\n<ul>\n<li>\u2018c\u2019 (regularization strength), \u2018penalty\u2019 and \u2018solver\u2019 in logistic regression<\/li>\n<li>\u2018learning rate\u2019, \u2018batch size\u2019, \u2018number of hidden layers\u2019 etc. in an artificial neural network<\/li>\n<\/ul>\n<p>The choice of Machine learning model depends on the dataset, the task in hand i.e. prediction or classification. Each model has its own unique set of hyperparameter and the task of finding the best combination of these parameter is known as hyperparameter optimization.<\/p>\n<p>For solving hyperparameter optimization problem there are various methods are available. For example :<\/p>\n<ul>\n<li>Grid Search<\/li>\n<li>Random Search<\/li>\n<li>Optuna<\/li>\n<li>HyperOpt<\/li>\n<\/ul>\n<p>In this post, we will focus on Optuna library which has one of the most accurate and successful hyperparameter optimization strategy.<\/p>\n<p><strong>Optuna :<\/strong><\/p>\n<p>Optuna is an open source hyperparameter optimization (HPO) framework to automate search space of hyperparameter. For finding an optimal set of hyperparameters, Optuna uses Bayesian method. It supports various types of samplers listed below :<\/p>\n<ul>\n<li>GridSampler(using grid search)<\/li>\n<li>RandomSampler(using random sampling)<\/li>\n<li>TPESampler\u00a0(using Tree-structured Parzen Estimator algorithm)<\/li>\n<li>CmaEsSampler\u00a0( using CMA-ES algorithm)<\/li>\n<\/ul>\n<p>Use of Optuna for hyperparameter optimization is explained using\u00a0<a href=\"https:\/\/www.kaggle.com\/mlg-ulb\/creditcardfraud\">Credit Card Fraud Detection<\/a>\u00a0dataset on Kaggle. The problem statement is to classify a credit card transaction fraudulent or genuine(binary classification). This data contains only numerical input variables which are PCA transformation of original features. Due to confidentially issues, the original features and more background information about the data are not available.<\/p>\n<p>In this case, we have used only a subset of the dataset to speed up the training time and to ensure the two different classes reach a perfectly balance. Here, the sampling method used is\u00a0TPESampler\u00a0. A subset of a dataset is shown in the figure below :<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28364 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-5-1.jpg\" alt=\"Efficient Hyperparam\" width=\"645\" height=\"183\" \/><\/p>\n<p>A subset of Credit Card Fraud Detection dataset<\/p>\n<p><strong>Importing required packages :<\/strong><\/p>\n<p>import optuna<br \/>\nfrom optuna import Trial, visualization<br \/>\nfrom optuna.samplers import TPESampler<br \/>\nfrom xgboost import XGBClassifier<\/p>\n<p><strong>Following are the main steps involved in HPO using Optuna for XGBoost model:<\/strong><\/p>\n<ol>\n<li><strong> Define Objective Function :<\/strong><br \/>\nThe first important step is to define an objective function. The objective should be to return a real value which has to minimize or maximize. In our case, we will be training XGBoost model and using the cross-validation score for evaluation. We will be returning this cross-validation score from an objective function which has to be maximized.<\/li>\n<li><strong> Define Hyperparameter Search Space :<br \/>\n<\/strong>Optuna supports five kind of hyperparameters distribution, which are given as follows :<\/li>\n<\/ol>\n<ul>\n<li><strong>Integer parameter<\/strong>: A uniform distribution on integers.<br \/>\nn_estimators = trial.suggest_int(&#8216;n_estimators&#8217;,100,500)<\/li>\n<li><strong>Categorical parameter<\/strong>: A categorical distribution.<br \/>\ncriterion = trial.suggest_categorical(&#8216;criterion&#8217; ,[&#8216;gini&#8217;, &#8216;entropy&#8217;])<\/li>\n<li><strong>Uniform parameter<\/strong>: A uniform distribution in linear domain.<br \/>\nsubsample = trial.suggest_uniform(&#8216;subsample&#8217; ,0.2,0.8)<\/li>\n<li><strong>Discrete-uniform parameter<\/strong>: A discretized uniform distribution in linear domain.<br \/>\nmax_features = trial.suggest_discrete_uniform(&#8216;max_features&#8217;, 0.05,1,0.05)<\/li>\n<li><strong>Loguniform parameter<\/strong>: A uniform distribution in log domain.<br \/>\nlearning_rate = trial.sugget_loguniform(&#8216;learning_rate&#8217; : 1e-6, 1e-3)<\/li>\n<\/ul>\n<p>The below figure shows the objective function and hyperparameter for our example.<\/p>\n<p>Objective Function<\/p>\n<ol start=\"3\">\n<li><strong> Study Objective :<br \/>\n<\/strong>We have to understand some important terminologies mentioned in their docs, which will make our work easier. These are given as follows :<\/li>\n<\/ol>\n<ul>\n<li><strong>Trial\u00a0<\/strong>: A single call of the objective function<\/li>\n<li><strong>Study\u00a0<\/strong>: An optimization session, which is a set of trails<\/li>\n<li><strong>Parameter\u00a0<\/strong>: A variable whose value is to be optimized such as value of \u201cn_estimators\u201d<\/li>\n<\/ul>\n<p>The Study object is used to manage optimization process. Method\u00a0<a href=\"https:\/\/optuna.readthedocs.io\/en\/stable\/reference\/study.html#optuna.study.create_study\">create_study()<\/a>\u00a0returns a study object. A study object has useful properties for analyzing the optimization outcome. In method of\u00a0create_study(), we have to define the direction of objective function i.e. \u201cmaximize\u201d or \u201cminimize\u201d and sampler for example\u00a0TPESampler(). After creating study, we can call\u00a0Optimize()<em>.<\/em><\/p>\n<ol start=\"4\">\n<li><strong> Best Trial and Result :<br \/>\n<\/strong>Once the optimization process completed then we can obtain the best parameters value and the optimal value of the objective function.<\/li>\n<\/ol>\n<p>Best trial: score 0.9427118644067797,<br \/>\nparams {&#8216;n_estimators&#8217;: 396, &#8216;max_depth&#8217;: 6, &#8216;reg_alpha&#8217;: 3, &#8216;reg_lambda&#8217;: 3, &#8216;min_child_weight&#8217;: 2, &#8216;gamma&#8217;: 0, &#8216;learning_rate&#8217;: 0.09041583301198859, &#8216;colsample_bytree&#8217;: 0.45999999999999996}<\/p>\n<ol start=\"5\">\n<li><strong> Trail History :<br \/>\n<\/strong>We can get the entire history of all the trial in the form data frame by just calling\u00a0study.trails_dataframe()<em>.<\/em><\/li>\n<\/ol>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28366 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-7-1.png\" alt=\"Efficient Hyperparam\" width=\"998\" height=\"266\" \/><\/p>\n<ol start=\"6\">\n<li><strong> Visualizations :<\/strong><\/li>\n<\/ol>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28368 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-9-1.jpg\" alt=\"Efficient Hyperparam\" width=\"1050\" height=\"700\" \/><\/p>\n<p>Photo by\u00a0<a href=\"https:\/\/unsplash.com\/@isaacmsmith?utm_source=medium&amp;utm_medium=referral\">Isaac Smith<\/a>\u00a0on\u00a0<a href=\"https:\/\/unsplash.com\/?utm_source=medium&amp;utm_medium=referral\">Unsplash<\/a><\/p>\n<p>Visualizing the hyperparameter search space can be very useful. From the visualization, we can gain some useful information on the interaction between parameters and we can see where to search next. The\u00a0<em>optuna.visualization\u00a0<\/em>module includes a set of useful visualizations.<\/p>\n<p>i)\u00a0plot_optimization_history(study): plots optimization history of all trials as well as the best score at each point.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28370 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-11-1.png\" alt=\"Efficient Hyperparam\" width=\"700\" height=\"450\" \/><\/p>\n<p>Optimization History Plot<\/p>\n<p>ii)\u00a0plot_slice(study): plots the parameter relationship as slice also we can see which part of search space were explored more.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28372 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-13-1.png\" alt=\"Efficient Hyperparam\" width=\"700\" height=\"450\" \/><\/p>\n<p>Slice Plot<\/p>\n<p>iii)\u00a0plot_parallel_coordinate(study)\u00a0:<em>\u00a0<\/em>plots the interactive visualization of the high-dimensional parameter relationship in study and scores.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28374 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-15-1.png\" alt=\"Efficient Hyperparam\" width=\"700\" height=\"450\" \/><\/p>\n<p>Parallel Coordinate Plot<\/p>\n<p>iv)\u00a0plot_contor(study): plots parameter interactive chart from we can choose which hyperparameter space has to explore.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"img-responsive alignnone wp-image-28376 size-full\" src=\"https:\/\/dev.enki.studio\/test\/wp-content\/uploads\/2022\/01\/Efficient-Hyperparam-17-1.png\" alt=\"Efficient Hyperparam\" width=\"700\" height=\"450\" \/><\/p>\n<p>Contour Plot<\/p>\n<p>Overall,\u00a0<strong>Visualizations are Amazing in Optuna !!<\/strong><\/p>\n<p><strong>Want to see AI\/ML models in action? <\/strong><\/p>\n<p style=\"text-align: center; font-size: 22px;\"><span style=\"color: #00188f;\"><strong>Build your first AI model for Free with our 14-day free trial of HyperSense AI studio<\/strong><\/span><\/p>\n<p style=\"text-align: center;\"><a class=\"cta-btn\" href=\"https:\/\/hypersense.subex.com\/ai-studio\/\" target=\"_blank\" rel=\"noopener noreferrer\">Get Free Trial<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction : Hyperparameter optimization is the science of tuning or choosing the best set of hyperparameters for a learning algorithm. A set of optimal hyperparameter has a big impact on the performance of any machine learning algorithm. It is one of the most time-consuming yet a crucial step in machine learning training pipeline. A Machine [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":28358,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[254],"tags":[],"class_list":["post-28356","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-augmented-analytics"],"acf":[],"_links":{"self":[{"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/posts\/28356","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=28356"}],"version-history":[{"count":0,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/posts\/28356\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=\/wp\/v2\/media\/28358"}],"wp:attachment":[{"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=28356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=28356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.enki.studio\/test\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=28356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}