{"id":8900,"date":"2018-08-09T17:30:58","date_gmt":"2018-08-09T17:30:58","guid":{"rendered":"https:\/\/www.uruit.com\/blog\/?p=8900"},"modified":"2023-06-01T01:35:21","modified_gmt":"2023-06-01T04:35:21","slug":"software-estimation-machine-learning","status":"publish","type":"post","link":"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/","title":{"rendered":"Use Machine Learning for Software Development Estimation"},"content":{"rendered":"<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_17 counter-hierarchy counter-decimal ez-toc-grey\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\">Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" style=\"display: none;\"><i class=\"ez-toc-glyphicon ez-toc-icon-toggle\"><\/i><\/a><\/span><\/div>\n<nav><ul class=\"ez-toc-list ez-toc-list-level-1\"><li class=\"ez-toc-page-1 ez-toc-heading-level-1\"><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Use_Machine_Learning_for_Software_Development_Estimation\" title=\"Use Machine Learning for Software Development Estimation\">Use Machine Learning for Software Development Estimation<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Generally_the_mental_process_that_each_team_member_goes_through_when_estimating_a_new_task_is\" title=\"Generally, the mental process that each team member goes through when estimating a new task is:\">Generally, the mental process that each team member goes through when estimating a new task is:<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Out1\" title=\"Out[1]:\">Out[1]:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Preparing_the_data\" title=\"Preparing the data\">Preparing the data<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_22\" title=\"In\u00a0[22]:\">In\u00a0[22]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_23\" title=\"In\u00a0[23]:\">In\u00a0[23]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Out_23\" title=\"Out [23]:\">Out [23]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_24\" title=\"In\u00a0[24]:\">In\u00a0[24]:<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Now_lets_see_how_our_data_looks_in_the_first_few_rows\" title=\"Now,\u00a0let\u2019s see how our data looks in the first few rows:\">Now,\u00a0let\u2019s see how our data looks in the first few rows:<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_25\" title=\"In\u00a0[25]:\">In\u00a0[25]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_26\" title=\"In\u00a0[26]:\">In\u00a0[26]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Out26\" title=\"Out[26]:\">Out[26]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_27\" title=\"In\u00a0[27]:\">In\u00a0[27]:<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Lets_see_the_amount_of_items_per_point\" title=\"Let&#8217;s see the amount of items per point:\">Let&#8217;s see the amount of items per point:<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_28\" title=\"In\u00a0[28]:\">In\u00a0[28]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Out28\" title=\"Out[28]:\">Out[28]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_29\" title=\"In\u00a0[29]:\">In\u00a0[29]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_30\" title=\"In\u00a0[30]:\">In\u00a0[30]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Out30\" title=\"Out[30]:\">Out[30]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Common_issues_generally_faced_during_the_data_preparation_phase\" title=\"Common issues generally faced during the data preparation phase:\">Common issues generally faced during the data preparation phase:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-21\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_31\" title=\"In\u00a0[31]:\">In\u00a0[31]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-22\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_32\" title=\"In\u00a0[32]:\">In\u00a0[32]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-23\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_33\" title=\"In\u00a0[33]:\">In\u00a0[33]:<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-24\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Dealing_with_the_imbalanced_dataset_-_Oversampling\" title=\"Dealing with the imbalanced dataset &#8211; Oversampling\">Dealing with the imbalanced dataset &#8211; Oversampling<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-25\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_34\" title=\"In\u00a0[34]:\">In\u00a0[34]:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-26\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Creating_our_classifier\" title=\"Creating our classifier\">Creating our classifier<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-27\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#FastText\" title=\"FastText\">FastText<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-28\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#_The_training_process_is_executed_with_the_following_parameters\" title=\"(*)\u00a0The training process is executed with the following parameters:\">(*)\u00a0The training process is executed with the following parameters:<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-29\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_35\" title=\"In\u00a0[35]:\">In\u00a0[35]:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-30\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#About_pre-trained_models\" title=\"About pre-trained models\">About pre-trained models<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-31\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Deep_Learning\" title=\"Deep Learning\">Deep Learning<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-32\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#The_code_below_shows_the_files_that_were_used_to_train_the_pre-trained_vectors_using_other_open_source_repositories_and_how_to_join_them_into_a_single_pandas_dataframe\" title=\"The code below shows the files that were used to train the pre-trained vectors using other open source repositories and how to join them into a single pandas dataframe.\">The code below shows the files that were used to train the pre-trained vectors using other open source repositories and how to join them into a single pandas dataframe.<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-33\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_36\" title=\"In\u00a0[36]:\">In\u00a0[36]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-34\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_16\" title=\"In\u00a0[16]:\">In\u00a0[16]:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-35\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_17\" title=\"In\u00a0[17]:\">In\u00a0[17]:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-36\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Defining_the_training_an_evaluation_strategy\" title=\"Defining the training: an evaluation strategy\">Defining the training: an evaluation strategy<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-37\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Selecting_Training_and_Testing_sets\" title=\"Selecting Training and Testing sets\">Selecting Training and Testing sets<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-38\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_37\" title=\"In [37]:\">In [37]:<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-39\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Defining_the_evaluation_criteria\" title=\"Defining the evaluation criteria\">Defining the evaluation criteria<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-40\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Confusion_matrix_precision_recall_and_F1_score_metrics\" title=\"Confusion matrix, precision, recall, and F1 score metrics\">Confusion matrix, precision, recall, and F1 score metrics<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-41\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Some_important_metrics_that_we_can_calculate_from_the_values_in_this_matrix_are\" title=\"Some important metrics that we can calculate from the values in this matrix are:\">Some important metrics that we can calculate from the values in this matrix are:<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-42\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_38\" title=\"In\u00a0[38]:\">In\u00a0[38]:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-43\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Main_method_k-fold_cross-validation_with_oversampling_in_training_folders\" title=\"Main method (k-fold cross-validation, with oversampling in training folders)\">Main method (k-fold cross-validation, with oversampling in training folders)<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-44\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#In_39\" title=\"In\u00a0[39]:\">In\u00a0[39]:<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-45\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Evaluating_the_results\" title=\"Evaluating the results\">Evaluating the results<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-46\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Class_0\" title=\"Class 0\">Class 0<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-47\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Class_1\" title=\"Class 1\">Class 1<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-48\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Class_2\" title=\"Class 2\">Class 2<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-49\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#About_precision\" title=\"About precision\">About precision<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-50\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#About_recall\" title=\"About recall\">About recall<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-51\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#About_the_F1_score\" title=\"About the F1 score\">About the F1 score<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-52\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Building_the_application\" title=\"Building the application\">Building the application<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-53\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Later_we_used_the_to_json_method_as_follows\" title=\"Later we used the \u201cto_json\u201d method as follows:\">Later we used the \u201cto_json\u201d method as follows:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-54\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Final_thoughts_on_software_development_estimation\" title=\"Final thoughts on software development estimation\">Final thoughts on software development estimation<\/a><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><ul class=\"ez-toc-list-level-5\"><li class=\"ez-toc-heading-level-5\"><a class=\"ez-toc-link ez-toc-heading-55\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#Some_of_the_important_things_that_we_can_conclude_from_this_work_are\" title=\"Some of the important things that we can conclude from this work are:\">Some of the important things that we can conclude from this work are:<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-56\" href=\"https:\/\/uruit.com\/blog\/software-estimation-machine-learning\/#About_us\" title=\"About us\">About us<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"Use_Machine_Learning_for_Software_Development_Estimation\"><\/span><span data-sheets-value=\"{&quot;1&quot;:2,&quot;2&quot;:&quot;Use Machine Learning for Software Development Estimation&quot;}\" data-sheets-userformat=\"{&quot;2&quot;:15231,&quot;3&quot;:{&quot;1&quot;:0},&quot;4&quot;:{&quot;1&quot;:2,&quot;2&quot;:16777215},&quot;5&quot;:{&quot;1&quot;:[{&quot;1&quot;:2,&quot;2&quot;:0,&quot;5&quot;:{&quot;1&quot;:2,&quot;2&quot;:0}},{&quot;1&quot;:0,&quot;2&quot;:0,&quot;3&quot;:3},{&quot;1&quot;:1,&quot;2&quot;:0,&quot;4&quot;:1}]},&quot;6&quot;:{&quot;1&quot;:[{&quot;1&quot;:2,&quot;2&quot;:0,&quot;5&quot;:{&quot;1&quot;:2,&quot;2&quot;:0}},{&quot;1&quot;:0,&quot;2&quot;:0,&quot;3&quot;:3},{&quot;1&quot;:1,&quot;2&quot;:0,&quot;4&quot;:1}]},&quot;7&quot;:{&quot;1&quot;:[{&quot;1&quot;:2,&quot;2&quot;:0,&quot;5&quot;:{&quot;1&quot;:2,&quot;2&quot;:0}},{&quot;1&quot;:0,&quot;2&quot;:0,&quot;3&quot;:3},{&quot;1&quot;:1,&quot;2&quot;:0,&quot;4&quot;:1}]},&quot;8&quot;:{&quot;1&quot;:[{&quot;1&quot;:2,&quot;2&quot;:0,&quot;5&quot;:{&quot;1&quot;:2,&quot;2&quot;:0}},{&quot;1&quot;:0,&quot;2&quot;:0,&quot;3&quot;:3},{&quot;1&quot;:1,&quot;2&quot;:0,&quot;4&quot;:1}]},&quot;9&quot;:0,&quot;11&quot;:3,&quot;12&quot;:0,&quot;14&quot;:{&quot;1&quot;:2,&quot;2&quot;:0},&quot;15&quot;:&quot;Calibri&quot;,&quot;16&quot;:11}\">Use Machine Learning for Software Development Estimation<\/span><span class=\"ez-toc-section-end\"><\/span><\/h1>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p><b>In this work, we will present some ideas on how to build a smart component that is able to predict the complexity of a software development task<\/b>. In particular, we will try to automate the process of sizing a task based on the information that is provided as part of its title and description and also leveraging the historical data of previous estimations.<\/p>\n<p>In agile development, this technique is known as story points estimation and it defers from other classic estimation techniques that predict hours because\u00a0<b>the goal of this software development estimation is not to estimate how long a task will take, but to predict how <u>complex<\/u>\u00a0a task is.<\/b><\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\"><center><img loading=\"lazy\" class=\"aligncenter wp-image-8901 size-full\" src=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414.png\" alt=\"\" width=\"1009\" height=\"667\" srcset=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414.png 1009w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414-300x198.png 300w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414-768x508.png 768w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414-750x496.png 750w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/maxresdefault_ed-e1533903732414-20x13.png 20w\" sizes=\"(max-width: 1009px) 100vw, 1009px\" \/><\/center><\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Typically, this <strong>software development estimation process is done by <a href=\"https:\/\/uruit.com\/blog\/going-back-to-agile-values\/\">agile teams<\/a> in order to know what all the tasks are that the team can commit to complete in the next sprint<\/strong>, generally a period of 2 weeks. Based on previous experience in the past 2 or 3 sprints, they can know in advance the average amount of points they are able to complete and so, they take that average as a measure of the threshold for the next sprint. Then, during the estimation process (generally through a fun activity like a planning poker), each member of the team gives a number of points that reflects how complex they think the task is.<\/p>\n<p>There is a set of tasks that the team has decided to be the\u00a0<i>&#8220;base histories&#8221;<\/i> that are well-known tasks already labeled with their associated complexity (that the team has agreed on) and can be used later as a base for comparison. After each sprint, new tasks can be added to this list. With time, this list can collect several examples of tasks with different complexities that the team can use later to compare and estimate new tasks. Teams generally reach a high level of accuracy in their estimations after some time, due to the continuous improvement based on accumulated experience, collecting more and more software development estimations.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Generally_the_mental_process_that_each_team_member_goes_through_when_estimating_a_new_task_is\"><\/span><b>Generally, the mental process that each team member goes through when estimating a new task is:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ul>\n<li>Based on his\/her previous experience, he\/she looks for similar tasks that they have done in the past<\/li>\n<li>He\/she gives the same number of points that similar tasks have been assigned in the past.<\/li>\n<li>If there isn\u2019t a similar task, he\/she starts an ordered comparison from the least to the most complex tasks. The reasoning is something like this:\u00a0<i>&#8220;Is this task more complex than these ones?&#8221;<\/i>\u00a0If so, he\/she moves on with the set of well-known base tasks in order of increasing complexity. This process is repeated until the new task falls in one of the categories of complexity. By convention, in case a new task looks more complex than size X but less complex than size Y (Y being the next size in order of complexity after X) the size assigned for the task&#8217;s estimation will be Y.<\/li>\n<\/ul>\n<p>If we look at this process,\u00a0<b>we can find lots of similarities with a classic machine learning problem<\/b>, where we have a task, T, that improves over time with experience, E, by a performance measure, P. In our case,\u00a0<b>T<\/b>\u00a0is the task of estimating\/predicting the complexity of a new ticket (bug, new feature, improvement, support, etc), the experience,\u00a0<b>E<\/b>, is the historical data of previous estimations and the performance measure, P, is the difference between the actual level of complexity and the software development estimation.<\/p>\n<p>In the following, we present a <a href=\"https:\/\/uruit.com\/machine-learning-consulting\">machine learning<\/a> approach to predict the complexity of a new task, based on the historical data of previously estimated tasks. We will use the\u00a0<b>Facebook FastText<\/b>\u00a0tool to learn text vector representations (word and sentence embeddings) that will be implicit used as input for a text classifier that classifies a task in three categories: easy, medium and complex. Note that we are changing things a bit, going from a point based estimation to a category based estimation.<\/p>\n<p>This is because unfortunately, the number of estimates were very\u00a0<b>unbalanced<\/b>\u00a0in our dataset, and by grouping tasks into these three categories, we can slightly simplify our problem. Anyway, we can think of each of these classes (easy, medium, complex) as points in a simplified version of the story points estimation process (in a real story points estimation, size generally follows a Fibonacci sequence 1, 2, 3, 5, 8, 13, 21, etc., or some minor variation of this sequence).<\/p>\n<p>In the end, we will build a basic web application like the one below, that is able to use the model that we trained so you can see it in action. It will allow you to search and pick up stories from the backlog (testing set) so you can compare the team&#8217;s average software development estimation vs the AI estimation (<b>to see the AI&#8217;s prediction vs the team&#8217;s, click the cards on the right side!\u00a0<\/b>). Sounds cool, yeah? Well, let\u2019s dive in!<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<h5 class=\"prompt output_prompt\"><span class=\"ez-toc-section\" id=\"Out1\"><\/span>Out[1]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"output_html rendered_html output_subarea output_execute_result\"><iframe loading=\"lazy\" width=\"100%\" height=\"430\" marginwidth=\"0\" marginheight=\"0\" hspace=\"0\" vspace=\"0\" frameborder=\"0\" src=\"https:\/\/waljoel.github.io\/story-points-prediction\/index.html\" data-mce-fragment=\"1\"><\/iframe><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div><\/div>\n<div class=\"prompt input_prompt\">Although it&#8217;s not required, in order to follow this work better, we recommend that you know some basic concepts around machine learning. Our free eBook,\u00a0<a href=\"https:\/\/uruit.com\/blog\/introduction-machine-learning-ebook\/\" class=\"broken_link\" rel=\"nofollow\"><i>The Business Executive\u2019s Guide to Smart Applications<\/i><\/a>, can give you a quick introduction to the topic if you need it. Also, our practical machine learning tutorial,\u00a0<a href=\"https:\/\/uruit.com\/blog\/soccer-and-machine-learning-tutorial\/\"><i>Soccer and Machine Learning: 2 hot topics for 2018<\/i><\/a>, can be a good resource for learning about a typicall machine learning workflow from scratch.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Having said that, now let\u2019s start!<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h3 id=\"Preparing-the-data\"><span class=\"ez-toc-section\" id=\"Preparing_the_data\"><\/span>Preparing the data<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">In order to train a neural network model for text classification we will use part of the dataset collected during the research presented in the paper,\u00a0<i><a href=\"https:\/\/arxiv.org\/pdf\/1609.00489\" class=\"external\" rel=\"nofollow\">A deep learning model for estimating story points<\/a><\/i>, that you can download from\u00a0<a href=\"https:\/\/github.com\/SEAnalytics\/datasets\/tree\/master\/storypoint\/IEEE%20TSE2018\/dataset\" class=\"broken_link external\" rel=\"nofollow\">this github repository<\/a><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p><b>Let&#8217;s start by loading the appceleratorstudio dataset<\/b><\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_22\"><\/span>In\u00a0[22]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import pandas as pd\r\nimport numpy as np\r\n\r\ndf = pd.read_csv(\"appceleratorstudio.csv\", usecols=['issuekey', 'title', 'description', 'storypoint'])<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Since our estimations will be based on the text information provided in the title and description columns, <b>let&#8217;s check for null or empty values in the dataset<\/b><\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_23\"><\/span>In\u00a0[23]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<\/div>\n<div class=\"input\">\n<div>\n<pre class=\"lang:python decode:true\">df.isnull().sum()<\/pre>\n<\/div>\n<h5><span class=\"ez-toc-section\" id=\"Out_23\"><\/span>Out [23]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">issuekey        0\r\ntitle           0\r\ndescription    43\r\nstorypoint      0\r\ndtype: int64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">We can see that there are 43 entries that have null values in the description column. So, <b>let&#8217;s remove any entry that is not complete:<\/b><\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_24\"><\/span>In\u00a0[24]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df = df.dropna(how='any')<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h4><span class=\"ez-toc-section\" id=\"Now_lets_see_how_our_data_looks_in_the_first_few_rows\"><\/span>Now,\u00a0<b>let\u2019s see how our data looks in the first few rows:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_25\"><\/span>In\u00a0[25]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df.head()\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[25]:<\/div>\n<div class=\"output_html rendered_html output_subarea output_execute_result\">\n<div>\n<table class=\"dataframe\" border=\"1\">\n<thead>\n<tr>\n<th><\/th>\n<th>issuekey<\/th>\n<th>title<\/th>\n<th>description<\/th>\n<th>storypoint<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<th>0<\/th>\n<td>TISTUD-6<\/td>\n<td>Add CA against object literals in function inv&#8230;<\/td>\n<td>{html}&lt;div&gt;&lt;p&gt;The idea here is that if our met&#8230;<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<th>1<\/th>\n<td>TISTUD-9<\/td>\n<td>Update branding for Appcelerator plugin to App&#8230;<\/td>\n<td>{html}&lt;div&gt;&lt;p&gt;At least fix feature icons, asso&#8230;<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<th>2<\/th>\n<td>TISTUD-11<\/td>\n<td>Create new JSON schema for SDK team<\/td>\n<td>{html}&lt;div&gt;&lt;p&gt;Create JSON schema containing pr&#8230;<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<th>3<\/th>\n<td>TISTUD-13<\/td>\n<td>Create Project References Property Page<\/td>\n<td>{html}&lt;div&gt;&lt;p&gt;Create property page for project&#8230;<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<th>4<\/th>\n<td>TISTUD-16<\/td>\n<td>New Desktop Project Wizard<\/td>\n<td>{html}&lt;div&gt;&lt;p&gt;Desktop (need to convert existin&#8230;<\/td>\n<td>1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>A very good approach is to\u00a0<b>take a look at the main characteristics of the data<\/b>\u00a0that you are going to be working on. In order to do this, we can use the\u00a0<i>describe<\/i>\u00a0operation available in any pandas dataframe:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_26\"><\/span>In\u00a0[26]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df.storypoint.describe()\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<h5 class=\"prompt output_prompt\"><span class=\"ez-toc-section\" id=\"Out26\"><\/span>Out[26]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>count    2876.000000\r\nmean        5.636300\r\nstd         3.309936\r\nmin         1.000000\r\n25%         3.000000\r\n50%         5.000000\r\n75%         8.000000\r\nmax        40.000000\r\nName: storypoint, dtype: float64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">As you can see above, by using the describe operation provided by any Pandas dataframe we can get a <b>summary of some important properties of our data<\/b>like:<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<ul>\n<li>the number of rows (a.k.a observations)<\/li>\n<li>average values<\/li>\n<li>minimums and maximums<\/li>\n<li>percentiles\u2019 values<\/li>\n<li>standard deviation<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Another good idea is to\u00a0<b>plot a histogram<\/b>. A histogram can give us a good notion of the underlying data distribution. What it basically does is split the possible values\/results into different bins and counts the number of occurrences (observations) where the variable under study falls into each bin. We can do this easily by using matplotlib, one of the most popular Python libraries for 2D plotting.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_27\"><\/span>In\u00a0[27]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import matplotlib.pyplot as plt\r\n \r\nplt.hist(df.storypoint, bins=20, alpha=0.6, color='y')\r\nplt.title(\"#Items per Point\")\r\nplt.xlabel(\"Points\")\r\nplt.ylabel(\"Count\")\r\n \r\nplt.show()\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt\"><\/div>\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAFx1JREFUeJzt3X2QZXV95\/H3R0YeFHV4GA3OYAaEnWioFXHEZwvBUiFWwJQSLLOOhM3gLhqRbARNxYdsZSMpV9SUpY6CYEJURC2IMRrkQWM2jM4g4iBMmCDCyDgzyKDiQxD87h\/313Bperrv6enue5t+v6pu3XN+53fO\/fYP+n7md87tc1NVSJI0qEcMuwBJ0vxicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0Na4JLcneTgYdeh+cPg0LyU5BtJDk1ycJJrxm27JcmL2\/Lrknx9OFXOnfZz3tdC4CdJrk3y8kH2raq9q+rmAV+nkhyya9VqvjM4NO8keSTwm8Am4BnANZPv8fCSZNFONv1bVe0NLAbOBS5Ksu\/cVaaFwuDQfHQY8N3q3fZgJTsJjiRPAT4MPKf9S\/yu1r5HkvckuTXJ1iQfTrJX23ZUks1J3pJkW5ItSU5IclySf09yZ5K39b3GkUnWtX\/lb03y3p3UMnbctyW5o82KXtO3fZCazkzyQ+Djkw1OVf0aOA\/YCzi4HeOPkmxq9V+a5Il9r33\/LCLJ+Uk+mOQfk\/w0ydokT27bvtZ2+XYbz9+frA49fBkcmjeSnNze\/P+VXhjcBfwJcHaSu5Ic1N+\/qm4AXk\/7l3hVLW6bzgb+C3A4cAiwFHh7366\/AezZ1\/5R4A\/ozW5eALy975rA+4H3V9VjgScDF03yI\/wGsH877ipgTZIVHWral95Ma\/UU47QI+O\/A3cBNSY4G\/go4ETgA+D7wqUkO8WrgXcA+9GZ1fwlQVS9s25\/WxvPTk9Whh7Gq8uFjXj2Af6H3Bvsk4Fog47bfAry4Lb8O+HrftgA\/A57c1\/Yc4Htt+SjgF8Bubf0xQAHP6uu\/HjihLX+N3pvs\/lPUfBRwL\/DovraLgD8fsKZ7gD0nOf7r2vHvAu4Aru4bg3OBv+7ruzfwK2B5Wy\/gkLZ8PvCxvr7HATf2rd\/f18fCfezsXKk0Utq5+pvpvcnuDVwF7NE270jyzqp63wCHWgI8Clif5P7DA7v19flRVd3Xln\/Rnrf2bf9FqwHgFOAvgBuTfA94V1V9YSevvaOqfta3\/n3giQPWtL2qfjnFz3Z1VT1\/gvYn0nc6r6ruTvIjerOaWybo\/8O+5Z\/zwM8qARgcmh+q6k5gcZKTgBdV1alJPg98sKq+Mtmu49bvoPfG\/9tV9YMZqOsm4NVJHgH8HnBxkv3GBcSYfZI8um\/bk4ANA9a0K7exvp3eKS4Akjwa2A\/Y5Z9fC5PXODTf9H+K6un0ThtNZiuwLMnucP+F448C5yR5PECSpUleOp1ikvxBkiXtuHe15vsm2eVdSXZP8gLg5cBnZrqmCfw9cHKSw5PsAfwfYG1V3TKNY22lXXDXwmVwaL55BnBNkv2A+6pqxxT9rwCuB36Y5I7Wdia9i75XJ\/kJ8BVgxU72n8rLgOuT3E3vQvlJk5xS+iGwg94M4ELg9VV14yzU9CBVdTm9aymfBbbQu4h\/0jQP907ggvZhhBNnoj7NP6nyi5yk2ZbkKODvqmrZsGuRdpUzDklSJwaHJKkTT1VJkjpxxiFJ6uRh+Xcc+++\/fy1fvnzYZUjSvLJ+\/fo7qmrJVP0elsGxfPly1q1bN+wyJGleSfL9Qfp5qkqS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1MnD8i\/H56uNG0+d9r4rVnxkBiuRpJ1zxiFJ6sTgkCR1YnBIkjoxOCRJnRgckqRODA5JUicGhySpE4NDktSJwSFJ6sTgkCR1YnBIkjoxOCRJnRgckqRODA5JUicGhySpE4NDktSJwSFJ6sTgkCR1YnBIkjqZteBIcl6SbUk29LXtm+SyJDe1531ae5J8IMmmJNclOaJvn1Wt\/01JVs1WvZKkwczmjON84GXj2s4CLq+qQ4HL2zrAscCh7bEa+BD0ggZ4B\/As4EjgHWNhI0kajlkLjqr6GnDnuObjgQva8gXACX3tn6ieq4HFSQ4AXgpcVlV3VtUO4DIeGkaSpDk019c4nlBVWwDa8+Nb+1Lgtr5+m1vbztofIsnqJOuSrNu+ffuMFy5J6hmVi+OZoK0maX9oY9WaqlpZVSuXLFkyo8VJkh4w18GxtZ2Coj1va+2bgQP7+i0Dbp+kXZI0JHMdHJcCY5+MWgVc0tf+2vbpqmcDP26nsr4MvCTJPu2i+EtamyRpSBbN1oGTfBI4Ctg\/yWZ6n456N3BRklOAW4FXte5fBI4DNgE\/B04GqKo7k\/xv4Jut319U1fgL7pKkOTRrwVFVr97JpmMm6FvAaTs5znnAeTNYmiRpF4zKxXFJ0jxhcEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE6GEhxJ3pzk+iQbknwyyZ5JDkqyNslNST6dZPfWd4+2vqltXz6MmiVJPXMeHEmWAn8MrKyqw4DdgJOAs4FzqupQYAdwStvlFGBHVR0CnNP6SZKGZFinqhYBeyVZBDwK2AIcDVzctl8AnNCWj2\/rtO3HJMkc1ipJ6jPnwVFVPwDeA9xKLzB+DKwH7qqqe1u3zcDStrwUuK3te2\/rv9\/44yZZnWRdknXbt2+f3R9CkhawYZyq2ofeLOIg4InAo4FjJ+haY7tMsu2Bhqo1VbWyqlYuWbJkpsqVJI0zjFNVLwa+V1Xbq+pXwOeA5wKL26krgGXA7W15M3AgQNv+OODOuS1ZkjRmGMFxK\/DsJI9q1yqOAb4LXAm8svVZBVzSli9t67TtV1TVQ2YckqS5MYxrHGvpXeS+BvhOq2ENcCZwRpJN9K5hnNt2ORfYr7WfAZw11zVLkh6waOouM6+q3gG8Y1zzzcCRE\/T9JfCquahLkjQ1\/3JcktSJwSFJ6sTgkCR1MpRrHJp5GzeeOu19V6z4yAxWIunhzhmHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwMFR5LnDdImSXr4G3TG8TcDtkmSHuYm\/erYJM8BngssSXJG36bHArvNZmGSpNE01XeO7w7s3fo9pq\/9J8ArZ6soSdLomjQ4quqrwFeTnF9V35+jmiRJI2yqGceYPZKsAZb371NVR89GUZKk0TVocHwG+DDwMeC+2StHkjTqBg2Oe6vqQzP1okkW0wuhw4AC\/hDYCHya3qzmFuDEqtqRJMD7geOAnwOvq6prZqoWSVI3g34c9x+S\/M8kByTZd+yxC6\/7fuBLVfVbwNOAG4CzgMur6lDg8rYOcCxwaHusBmYswCRJ3Q0641jVnv+0r62Ag7u+YJLHAi8EXgdQVfcA9yQ5HjiqdbsAuAo4Ezge+ERVFXB1ksVJDqiqLV1fW5K06wYKjqo6aAZf82BgO\/DxJE8D1gNvAp4wFgZVtSXJ41v\/pcBtfftvbm0PCo4kq+nNSHjSk540g+VKkvoNFBxJXjtRe1V9YpqveQTwxqpam+T9PHBaasKXn+ilJ6hlDbAGYOXKlQ\/ZLkmaGYOeqnpm3\/KewDHANcB0gmMzsLmq1rb1i+kFx9axU1BJDgC29fU\/sG\/\/ZcDt03hdSdIMGPRU1Rv715M8Dvjb6bxgVf0wyW1JVlTVRnoh9N32WAW8uz1f0na5FHhDkk8BzwJ+7PUNSRqeQWcc4\/2c3qecpuuNwIVJdgduBk6m9wmvi5KcAtwKvKr1\/SK9j+Juaq978i68riRpFw16jeMfeOC6wm7AU4CLpvuiVXUtsHKCTcdM0LeA06b7WpKkmTXojOM9fcv3At+vqs2zUM+8t3HjqcMuQZJm1UB\/ANhudngjvTvk7gPcM5tFSZJG16DfAHgi8A161x1OBNYm8bbqkrQADXqq6s+AZ1bVNoAkS4Cv0PsorSRpARn0XlWPGAuN5kcd9pUkPYwMOuP4UpIvA59s679P72OykqQFZqrvHD+E3j2k\/jTJ7wHPp3cLkH8DLpyD+iRJI2aq003vA34KUFWfq6ozqurN9GYb75vt4iRJo2eq4FheVdeNb6yqdfS+cEmStMBMFRx7TrJtr5ksRJI0P0wVHN9M8kfjG9v9pNbPTkmSpFE21aeqTgc+n+Q1PBAUK4HdgVfMZmGSpNE0aXBU1VbguUleBBzWmv+xqq6Y9cokSSNp0O\/juBK4cpZrkSTNA\/71tySpE4NDktSJwSFJ6sTgkCR1YnBIkjoxOCRJnRgckqRODA5JUicGhySpE4NDktSJwSFJ6sTgkCR1YnBIkjoxOCRJnQwtOJLsluRbSb7Q1g9KsjbJTUk+nWT31r5HW9\/Uti8fVs2SpOHOON4E3NC3fjZwTlUdCuwATmntpwA7quoQ4JzWT5I0JEMJjiTLgN8BPtbWAxwNXNy6XACc0JaPb+u07ce0\/pKkIRjWjON9wFuAX7f1\/YC7quretr4ZWNqWlwK3AbTtP279HyTJ6iTrkqzbvn37bNYuSQvanAdHkpcD26pqfX\/zBF1rgG0PNFStqaqVVbVyyZIlM1CpJGkiA33n+Ax7HvC7SY4D9gQeS28GsjjJojarWAbc3vpvBg4ENidZBDwOuHPuy5YkwRBmHFX11qpaVlXLgZOAK6rqNcCVwCtbt1XAJW350rZO235FVT1kxiFJmhuj9HccZwJnJNlE7xrGua39XGC\/1n4GcNaQ6pMkMZxTVferqquAq9ryzcCRE\/T5JfCqOS1MkrRTozTjkCTNAwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqZNGwCxhFGzeeOuwSJGlkOeOQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1MmcB0eSA5NcmeSGJNcneVNr3zfJZUluas\/7tPYk+UCSTUmuS3LEXNcsSXrAMGYc9wJ\/UlVPAZ4NnJbkqcBZwOVVdShweVsHOBY4tD1WAx+a+5IlSWPmPDiqaktVXdOWfwrcACwFjgcuaN0uAE5oy8cDn6ieq4HFSQ6Y47IlSc1Qr3EkWQ48HVgLPKGqtkAvXIDHt25Lgdv6dtvc2sYfa3WSdUnWbd++fTbLlqQFbWjBkWRv4LPA6VX1k8m6TtBWD2moWlNVK6tq5ZIlS2aqTEnSOEMJjiSPpBcaF1bV51rz1rFTUO15W2vfDBzYt\/sy4Pa5qlWS9GDD+FRVgHOBG6rqvX2bLgVWteVVwCV97a9tn656NvDjsVNakqS5N4zbqj8P+G\/Ad5Jc29reBrwbuCjJKcCtwKvati8CxwGbgJ8DJ89tuZKkfnMeHFX1dSa+bgFwzAT9CzhtVouSJA3MvxyXJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKmTYfwdh3S\/jRtPnfa+K1Z8ZAYrkTQoZxySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1InBIUnqxOCQJHVicEiSOjE4JEmdGBySpE4MDklSJwaHJKkTg0OS1MmiYRcgzUcbN5467X1XrPjIDFYizb15M+NI8rIkG5NsSnLWsOuRpIVqXgRHkt2ADwLHAk8FXp3kqcOtSpIWpvlyqupIYFNV3QyQ5FPA8cB3h1qVNA2e5tJURv3\/kfkSHEuB2\/rWNwPP6u+QZDWwuq3enWTjJMfbH7hjRiucOUOobc2gHUds3B5U94jV9iAzWNvA\/60GtUDGbcaNcG1rdqW23xyk03wJjkzQVg9aqVrDgL9VSdZV1cqZKGymWdv0WNv0WNv0LPTa5sU1DnozjAP71pcBtw+pFkla0OZLcHwTODTJQUl2B04CLh1yTZK0IM2LU1VVdW+SNwBfBnYDzquq63fhkDN+ongGWdv0WNv0WNv0LOjaUlVT95IkqZkvp6okSSPC4JAkdbKggmOUb1uS5JYk30lybZJ1I1DPeUm2JdnQ17ZvksuS3NSe9xmh2t6Z5Adt\/K5NctwQ6jowyZVJbkhyfZI3tfahj9sktY3CuO2Z5BtJvt1qe1drPyjJ2jZun24fjBmV2s5P8r2+cTt8rmvrq3G3JN9K8oW2PvvjVlUL4kHvovp\/AAcDuwPfBp467Lr66rsF2H\/YdfTV80LgCGBDX9tfA2e15bOAs0eotncC\/2vIY3YAcERbfgzw7\/RukTP0cZuktlEYtwB7t+VHAmuBZwMXASe19g8D\/2OEajsfeOUwx62vxjOAvwe+0NZnfdwW0ozj\/tuWVNU9wNhtSzSBqvoacOe45uOBC9ryBcAJc1pUs5Pahq6qtlTVNW35p8AN9O56MPRxm6S2oaueu9vqI9ujgKOBi1v7sMZtZ7WNhCTLgN8BPtbWwxyM20IKjoluWzISvzhNAf+cZH27fcooekJVbYHeGxHw+CHXM94bklzXTmUN5TTamCTLgafT+xfqSI3buNpgBMatnW65FtgGXEbv7MBdVXVv6zK039fxtVXV2Lj9ZRu3c5LsMYzagPcBbwF+3db3Yw7GbSEFx5S3LRmy51XVEfTuAHxakhcOu6B55kPAk4HDgS3A\/x1WIUn2Bj4LnF5VPxlWHROZoLaRGLequq+qDqd3V4gjgadM1G1uq2ovOq62JIcBbwV+C3gmsC9w5lzXleTlwLaqWt\/fPEHXGR+3hRQcI33bkqq6vT1vAz5P75dn1GxNcgBAe9425HruV1Vb2y\/4r4GPMqTxS\/JIem\/MF1bV51rzSIzbRLWNyriNqaq7gKvoXUdYnGTsj5SH\/vvaV9vL2qm\/qqr\/BD7OcMbtecDvJrmF3qn3o+nNQGZ93BZScIzsbUuSPDrJY8aWgZcAGybfayguBVa15VXAJUOs5UHG3pibVzCE8Wvnl88Fbqiq9\/ZtGvq47ay2ERm3JUkWt+W9gBfTuwZzJfDK1m1Y4zZRbTf2\/UMg9K4hzPm4VdVbq2pZVS2n9352RVW9hrkYt2F\/ImAuH8Bx9D5N8h\/Anw27nr66Dqb3Ka9vA9ePQm3AJ+mduvgVvdnaKfTOn14O3NSe9x2h2v4W+A5wHb036gOGUNfz6Z0WuA64tj2OG4Vxm6S2URi3\/wp8q9WwAXh7az8Y+AawCfgMsMcI1XZFG7cNwN\/RPnk1rAdwFA98qmrWx81bjkiSOllIp6okSTPA4JAkdWJwSJI6MTgkSZ0YHJKkTgwOaZqS3NfujLohyWeSPGqK\/v9vgGOePtVxpGEzOKTp+0VVHV5VhwH3AK+frHNVPXeAY54OGBwaaQaHNDP+BTgEIMkZbRayIcnpYx2S3N2ej0pyVZKLk9yY5ML0\/DHwRODK9t0Zu7XvfdiQ3ne1vHkoP5k0zqKpu0iaTLsv0LHAl5I8AzgZeBa9G86tTfLVqvrWuN2eDvw2vfsI\/Su9m1x+IMkZwIuq6o52rKVtRsPYrS+kYXPGIU3fXu122+uAW+ndC+r5wOer6mfV+x6HzwEvmGDfb1TV5urdXPBaYPkEfW4GDk7yN0leBozUnXa1cDnjkKbvF9W73fb92k3vBvGffcv3McHvYlXtSPI04KXAacCJwB9Os1ZpxjjjkGbW14ATkjyq3en4FfSufwzqp\/S+2pUk+wOPqKrPAn9O7+typaFzxiHNoKq6Jsn59O5OCvCxCa5vTGYN8E9JttD7hNXHk4z9A++tM1epNH3eHVeS1ImnqiRJnRgckqRODA5JUicGhySpE4NDktSJwSFJ6sTgkCR18v8B70UtS5NbwAAAAAAASUVORK5CYII=\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>We can easily see that the number of occurrences is not uniform throughout the different size categories (points).<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Lets_see_the_amount_of_items_per_point\"><\/span><b>Let&#8217;s see the amount of items per point:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_28\"><\/span>In\u00a0[28]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df.groupby('storypoint').size()\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<h5 class=\"prompt output_prompt\"><span class=\"ez-toc-section\" id=\"Out28\"><\/span>Out[28]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>storypoint\r\n1      148\r\n2      112\r\n3      571\r\n5     1126\r\n8      751\r\n9        1\r\n13     137\r\n20      22\r\n21       3\r\n34       1\r\n40       4\r\ndtype: int64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">By seeing the histogram and the number of examples per point, we can see that we have many more examples of 5 and 8 than the others. <b>That&#8217;s known as an\u00a0<u>imbalanced dataset<\/u>\u00a0and it could be an issue in classification problems<\/b>. There are different techniques to deal with imbalanced data, starting by collecting more samples of the entries with low frequency or generating new artificial entries (oversampling) or removing some entries in the classes with higher frequency (downsampling).<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>In our case, we will start by\u00a0<b>grouping points into three different categories<\/b>\u00a0to reduce the imbalanced data.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_29\"><\/span>In\u00a0[29]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df.loc[df.storypoint &lt;= 2, 'storypoint'] = 0 #small\r\ndf.loc[(df.storypoint &gt; 2) &amp; (df.storypoint &lt;= 5), 'storypoint'] = 1 #medium\r\ndf.loc[df.storypoint &gt; 5, 'storypoint'] = 2 #big<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_30\"><\/span>In\u00a0[30]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df.groupby('storypoint').size()\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<h5 class=\"prompt output_prompt\"><span class=\"ez-toc-section\" id=\"Out30\"><\/span>Out[30]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>storypoint\r\n0     260\r\n1    1697\r\n2     919\r\ndtype: int64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><b>After grouping we continue having imbalanced data, so we will do a basic oversampling and downsampling<\/b>, but we will do it later when we apply a\u00a0<b>cross-validation technique<\/b>\u00a0that is an approach for training and evaluating the performance of a machine learning model through different partitions of our dataset as we will explain later.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>At this point, it&#8217;s important to note that\u00a0<b>in this work the goal is to solve a classification problem<\/b>\u00a0(predict the class associated to the complexity of a task: 0-easy, 1-medium or 2-complex)\u00a0<b>instead of a regression problem<\/b>\u00a0(predict a continuous real value) like in the paper,\u00a0<i><a href=\"https:\/\/arxiv.org\/pdf\/1609.00489\" class=\"external\" rel=\"nofollow\">A deep learning model for estimating story points<\/a><\/i>.<\/p>\n<p>Before we continue, let&#8217;s do some\u00a0<b>cleanup<\/b>\u00a0to our data. This is also a common step that generally any machine learning process needs to apply because of the following issues:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"Common_issues_generally_faced_during_the_data_preparation_phase\"><\/span><b>Common issues generally faced during the data preparation phase:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<ul>\n<li>Format and structure normalization<\/li>\n<li>Detect and fix missing values<\/li>\n<li>Remove duplicates<\/li>\n<li>Normalize units<\/li>\n<li>Validate constraints<\/li>\n<li>Detect and remove anomalies<\/li>\n<li>Study features importance\/relevance<\/li>\n<li>Dimentional reduction, feature selection &amp; extraction<\/li>\n<\/ul>\n<p>For this work, most of these issues were already addressed by the authors of\u00a0<i><a href=\"https:\/\/arxiv.org\/pdf\/1609.00489\" class=\"external\" rel=\"nofollow\">A deep learning model for estimating story points<\/a><\/i>\u00a0when collecting the dataset. Anyway, we will need to do some extra cleanup to the data for our purpose:\u00a0<b>remove some html tags as well English stop words<\/b>\u00a0(words like\u00a0<i>the<\/i>,\u00a0<i>this<\/i>,\u00a0<i>that<\/i>, etc) because they can add noise to our problem and it&#8217;s better to remove them.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_31\"><\/span>In\u00a0[31]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import numpy as np\r\nimport csv\r\nfrom nltk.corpus import stopwords\r\n\r\n#Define some known html tokens that appear in the data to be removed later\r\nhtmltokens = ['{html}','&lt;div&gt;','&lt;pre&gt;','&lt;p&gt;', '&lt;\/div&gt;','&lt;\/pre&gt;','&lt;\/p&gt;']\r\n\r\n#Clean operation\r\n#Remove english stop words and html tokens\r\ndef cleanData(text):\r\n    \r\n    result = ''\r\n    \r\n    for w in htmltokens:\r\n        text = text.replace(w, '')\r\n    \r\n    text_words = text.split()    \r\n    \r\n    resultwords  = [word for word in text_words if word not in stopwords.words('english')]\r\n    \r\n    if len(resultwords) &gt; 0:\r\n        result = ' '.join(resultwords)\r\n    else:\r\n        print('Empty transformation for: ' + text)\r\n        \r\n    return result\r\n\r\ndef formatFastTextClassifier(label):\r\n    return \"__label__\" + str(label) + \" \"<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p><b>Important:<\/b>\u00a0Since we are removing stop words and html tags in our dataset, later when we want to predict with some unseen data we will need to apply the same transformation before requesting the model\u2019s prediction for that input.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">In order to work easily, <b>we will also create two new columns:<\/b><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<ul>\n<li>One new column called &#8220;title_desc&#8221; that is just the concatenation of the title and description columns<\/li>\n<li>A second column called &#8220;label_title_desc&#8221; that contains the number of points with a specific prefix expected by FastText to recognize it as the labeled information (class)<\/li>\n<\/ul>\n<p>While doing this, we will also change everything to\u00a0<b>lower case<\/b>\u00a0to make the training phase case insensitive. These new columns will be used later for training our learning algorithms.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_32\"><\/span>In\u00a0[32]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df['title_desc'] = df['title'].str.lower() + ' - ' + df['description'].str.lower()\r\ndf['label_title_desc'] = df['storypoint'].apply(lambda x: formatFastTextClassifier(x)) + df['title_desc'].apply(lambda x: cleanData(str(x)))<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Finally, since we were removing some empty entries, we will re-index our dataset to fix it and have continuous indices again:<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_33\"><\/span>In\u00a0[33]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">df = df.reset_index(drop=True)\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<h4 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"Dealing_with_the_imbalanced_dataset_-_Oversampling\"><\/span>Dealing with the imbalanced dataset &#8211; Oversampling<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<div class=\"prompt input_prompt\">As you will see in the final main method, in order to deal with imbalanced dataset we are doing a basic oversampling that simply consists of adding copies of the datapoints in the minority class until reaching the number of items in the majority class.<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Other more complex oversampling techniques exist, like SMOTE, where artificial datapoints (called synthetic data points) are created by taking two datapoints in the minority class (one datapoint and one of its k nearest neighbors) creating the new artificial point in the space between the two real points. If we think about this technique in a 2D scenario, the new datapoint is created in some random place on the line between the two points as you can see in the image below:<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8903\" aria-describedby=\"caption-attachment-8903\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" class=\"wp-image-8903 size-full\" src=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote.png\" alt=\"\" width=\"833\" height=\"326\" srcset=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote.png 833w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote-300x117.png 300w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote-768x301.png 768w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote-750x294.png 750w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/smote-20x8.png 20w\" sizes=\"(max-width: 833px) 100vw, 833px\" \/><figcaption id=\"caption-attachment-8903\" class=\"wp-caption-text\">Smote Oversampling (ref: http:\/\/rikunert.com\/SMOTE_explained)<\/figcaption><\/figure>\n<p><center><\/center>If you want to try more complex techniques you can try Python\u2019s excellent package,\u00a0<a href=\"https:\/\/github.com\/scikit-learn-contrib\/imbalanced-learn\" class=\"external\" rel=\"nofollow\">imbalanced-learn<\/a>\u00a0with several algorithms already implemented for you.<\/p>\n<p>Anyway, for this work, a basic oversampling technique that creates copies of the existing data was used. The main reason for that is simplicity, because dealing with new synthetic datapoint created artificially implies finding a text representation for a sentence that could map to that new vector representation, because in the end, the FastText tool expects text in sentences and not the embeddings. Possible workarounds exist for this like approximating the synthetic point with a new sentence generated by averaging the embeddings of words used by the nearest k sentences to the synthetic points, for instance. This could be something interesting to try, so if you do it please let us know your results!<\/p>\n<p><b>Note:<\/b>\u00a0Basic random downsampling of the majority class that is also a common and simple technique was combined with the oversampling, but didn\u2019t improve the results. So, in the end, just a basic oversampling was used in order to minimize the effect of an imbalanced dataset.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_34\"><\/span>In\u00a0[34]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">from collections import Counter\r\n\r\ndef SimpleOverSample(_xtrain, _ytrain):\r\n    xtrain = list(_xtrain)\r\n    ytrain = list(_ytrain)\r\n\r\n    samples_counter = Counter(ytrain)\r\n    max_samples = sorted(samples_counter.values(), reverse=True)[0]\r\n    for sc in samples_counter:\r\n        init_samples = samples_counter[sc]\r\n        samples_to_add = max_samples - init_samples\r\n        if samples_to_add &gt; 0:\r\n            #collect indices to oversample for the current class\r\n            index = list()\r\n            for i in range(len(ytrain)):\r\n                if(ytrain[i] == sc):\r\n                    index.append(i)\r\n            #select samples to copy for the current class    \r\n            copy_from = [xtrain[i] for i in index]\r\n            index_copy = 0\r\n            for i in range(samples_to_add):\r\n                xtrain.append(copy_from[index_copy % len(copy_from)])\r\n                ytrain.append(sc)\r\n                index_copy += 1\r\n    return xtrain, ytrain<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Before ending this section, I\u2019d like to highlight that sometimes people underestimate the data preparation stage (mainly those who are just starting with their first machine learning project) but you should know that this innocent-seeming first stage, most of the time, takes more than half of the total project time (sometimes even up to 60-80%!). So, keep that in mind and treat this stage with the importance it deserves.<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h3 id=\"Creating-our-classifier\"><span class=\"ez-toc-section\" id=\"Creating_our_classifier\"><\/span>Creating our classifier<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">In this work, we want to give\u00a0<a href=\"https:\/\/arxiv.org\/pdf\/1607.01759.pdf\" class=\"external\" rel=\"nofollow\">FastText<\/a>\u00a0a try, a tool developed by Facebook that is an extension of the well-known\u00a0<a href=\"https:\/\/arxiv.org\/pdf\/1301.3781.pdf\" class=\"external\" rel=\"nofollow\">Word2Vec<\/a>\u00a0word embedding tool previously created by a research team led by\u00a0<i>Tomas Mikolov<\/i>\u00a0at Google.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Without entering into much detail, we can say that\u00a0<b>embeddings consist of techniques that learn vector representations of words, sentences, or documents, so that the vector representation of similar and semantically related words, sentences, or documents are close together in the high dimensional vector space.<\/b>\u00a0Then, by leveraging this characteristic onto the learned vectors, we can use them as features for any kind of machine learning algorithm, to train a classifier or as input to a clustering algorithm, etc.<\/p>\n<p>Word embedding techniques are not something new in Natural Language Processing (NLP), although in the last years, new embeddings techniques based in predictive neural networks models have become very popular and they have revolutionized the machine learning field in many domains, not just that of NLP. Word embeddings have started to be used in other domains like e-commerce and recommender systems with the variation known as prod2vec, meta-prod2vec, or in mobile applications like app2vec, among others. Recently, I&#8217;ve applied different embedding techniques to create Internet Domain Name embeddings from DNS trace logs and they have demonstrated to be a good approach for learning semantic similarities and analogy tasks between Internet Domain Names. You can see details about how I have used word2vec for learning Internet Domain Names in\u00a0<a href=\"http:\/\/www.clei2017-46jaiio.sadio.org.ar\/sites\/default\/files\/Mem\/SLIOIA\/slioia-11.pdf\" class=\"broken_link external\" rel=\"nofollow\">Vector representation of Internet Domain Names using a Word Embedding technique<\/a><\/p>\n<h3><span class=\"ez-toc-section\" id=\"FastText\"><\/span>FastText<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In regard to FastText, its main advantage over word2vec is that it considers subwords inside a word. Instead of considering each word as a single token, a word is split into a set of substrings called ngrams and later the training phase is done considering each subword of a word, and the vector representation of a word is formed by averaging the vector representation of its subwords (and the word itself). The most important parameters to tune when using FastText are\u00a0<b>minn\u00a0<\/b>and\u00a0<b>maxn<\/b>\u00a0that define the min and max length of n-grams when splitting words.<\/p>\n<p>Additionally,\u00a0<b>FastText can be used either in supervised or unsupervised mode<\/b>. When using FastText in supervised mode, you can train a supervised model by using a dataset specially prepared (labeled) using a set of sentences (one per line) along with a label that acts as the class to which the sentence belongs. So, by training a FastText model in supervised mode, you can later perform classification tasks over new unseen sentences, which is very helpful for a lot of text classification and sentiment analysis problems.<\/p>\n<p>Having said this, we present a simple custom python wrapper for the supervised mode of the native FastText interface. Although there is already a wrapper for FastText and a native module in the well-known Gensim package, none of them include support for the supervised mode of FastText (only the unsupervised mode). So, I decided to create our custom and very basic wrapper with the minimum that we need for our purpose, that is:<\/p>\n<ul>\n<li>a constructor to create new instances of the wrapper with its own state<\/li>\n<li>a fit method to trigger the training process by calling the executable file and passing the required parameters (*)<\/li>\n<li>a predict method that receives an array with a list of sentences and returns another array (of the same size) with the integer predictions in {0, 1, 2} for each sentence<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"_The_training_process_is_executed_with_the_following_parameters\"><\/span><b>(*)<\/b>\u00a0The training process is executed with the following parameters:<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ul>\n<li>500 epochs (iterations over the corpus)<\/li>\n<li>Vector size of 300 dimensions<\/li>\n<li>minn=4 and maxn=6 (minimum and maximum numbers of n-grams respectively)<\/li>\n<li>pretrained file used to transfer previous knowledge of the English language and domain-specific knowledge (I&#8217;ve tried using the\u00a0pretrained vectors for English language\u00a0provided by FastText, but in the end generating my own pretrained models using other system datasets in the same domain worked better. You can download\u00a0<a href=\"https:\/\/github.com\/SEAnalytics\/datasets\/tree\/master\/storypoint\/IEEE%20TSE2018\/pretrain%20data\" class=\"broken_link external\" rel=\"nofollow\">these other datasets<\/a>\u00a0from the same github repository in order to use them for building your own pre-trained model.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_35\"><\/span>In\u00a0[35]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import uuid\r\nimport subprocess\r\n\r\nclass FastTextClassifier:\r\n\r\n    rand = \"\"\r\n    inputFileName = \"\"\r\n    outputFileName = \"\"\r\n    testFileName = \"\"\r\n    \r\n    def __init__(self):\r\n        self.rand = str(uuid.uuid4())\r\n        self.inputFileName = \"issues_train_\" + self.rand + \".txt\"\r\n        self.outputFileName = \"supervised_classifier_model_\" + self.rand\r\n        self.testFileName = \"issues_test_\" + self.rand + \".txt\"\r\n    \r\n    def fit(self, xtrain, ytrain):\r\n        outfile=open(self.inputFileName, mode=\"w\", encoding=\"utf-8\")\r\n        for i in range(len(xtrain)):\r\n            #line = \"__label__\" + str(ytrain[i]) + \" \" + xtrain[i]\r\n            line = xtrain[i]\r\n            outfile.write(line + '\\n')\r\n        outfile.close()            \r\n        p1 = subprocess.Popen([\"cmd\", \"\/C\", \"fasttext supervised -input \" + self.inputFileName + \" -output \" + self.outputFileName + \" -epoch 500 -wordNgrams 4 -dim 300 -minn 4 -maxn 6 -pretrainedVectors pretrain_model.vec\"],stdout=subprocess.PIPE)\r\n        p1.communicate()[0].decode(\"utf-8\").split(\"\\r\\n\")\r\n        \r\n        \r\n    def predict(self, xtest):\r\n        #save test file\r\n        outfile=open(self.testFileName, mode=\"w\", encoding=\"utf-8\")\r\n        for i in range(len(xtest)):\r\n            outfile.write(xtest[i] + '\\n')\r\n        outfile.close()\r\n        #get predictions\r\n        p1 = subprocess.Popen([\"cmd\", \"\/C\", \"fasttext predict \" + self.outputFileName + \".bin \" + self.testFileName],stdout=subprocess.PIPE)\r\n        output_lines = p1.communicate()[0].decode(\"utf-8\").split(\"\\r\\n\")\r\n        test_pred = [int(p.replace('__label__','')) for p in output_lines if p != '']\r\n        return test_pred<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h3 id=\"About-pre-trained-models\"><span class=\"ez-toc-section\" id=\"About_pre-trained_models\"><\/span>About pre-trained models<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Pre-trained models are generally used for transferring previous knowledge from a similar domain to our specific problem, where we can continue and extend the learning by specializing the model to our concrete problem. Using pre-trained models allows us to avoid starting from scratch and to also get good results even if we have small datasets for training.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>In our particular case, we base our predictions on understanding the meaning of the natural language information provided in the title and description of each entry. So, having a pretrained model of English text could be very helpful.<\/p>\n<p>My first attempt was to use the pre-trained model available in the\u00a0FastText Github repository\u00a0for the English language. Although this improved the overall accuracy of my solution a bit, the improvement was not considered too much.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Deep_Learning\"><\/span>Deep Learning<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The second approach that I followed was to use the same idea already applied in\u00a0<i><a href=\"https:\/\/arxiv.org\/pdf\/1609.00489\" class=\"external\" rel=\"nofollow\">A deep learning model for estimating story points<\/a><\/i>\u00a0and use the\u00a0<a href=\"https:\/\/github.com\/SEAnalytics\/datasets\/tree\/master\/storypoint\/IEEE%20TSE2018\/pretrain%20data\" class=\"broken_link external\" rel=\"nofollow\">pre-trained csv files<\/a>\u00a0(issues with title and description but without story points) in order to train a fastText unsupervised model in the same domain as my problem (software development issues) and use it later when training the fastText supervised classifier in my concrete problem and specific dataset. By doing this we can start with some basic knowledge, like vector representations for words and sentences in the software development domain (domain specific), so we can have a good parameter initialization without using the labeled data.<\/p>\n<p>Using the second approach was much better than the first one (final results will be shown at the end). Although the pre-trained vectors trained from Wikipedia had been trained with considerably much more data than the pre-trained vectors that I got using the pre-trained csv files for other open source repositories, the second approach achieved the best results. This confirms something already known, that the use of pre-trained vectors trained in a similar domain can achieve better results even if they&#8217;re trained with much fewer data than other pre-trained vectors that were trained in a less similar domain.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h4><span class=\"ez-toc-section\" id=\"The_code_below_shows_the_files_that_were_used_to_train_the_pre-trained_vectors_using_other_open_source_repositories_and_how_to_join_them_into_a_single_pandas_dataframe\"><\/span>The code below shows the files that were used to train the pre-trained vectors using other open source repositories and how to join them into a single pandas dataframe.<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_36\"><\/span>In\u00a0[36]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import pandas as pd\r\nimport numpy as np\r\n\r\npretrain_files = ['apache_pretrain.csv', \r\n                  'jira_pretrain.csv', \r\n                  'spring_pretrain.csv', \r\n                  'talendforge_pretrain.csv', \r\n                  'moodle_pretrain.csv',\r\n                  'appcelerator_pretrain.csv',\r\n                  'duraspace_pretrain.csv',\r\n                  'mulesoft_pretrain.csv',\r\n                  'lsstcorp_pretrain.csv']\r\n\r\npretrained = None\r\n\r\nfor file in pretrain_files:\r\n    df_pretrain = pd.read_csv('PretrainData\/' + file, usecols=['issuekey', 'title', 'description'])\r\n    if(pretrained is not None):\r\n        pretrained = pd.concat([pretrained, df_pretrain])\r\n    else:\r\n        pretrained = df_pretrain\r\n\r\npretrained = pretrained.dropna(how='any')<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">We also create a new column concatenating the title and description and applying the same cleanup method than before (to be consistent):<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_16\"><\/span>In\u00a0[16]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">pretrained['title_desc'] = (pretrained['title'].str.lower() + ' - ' + pretrained['description'].str.lower()).apply(lambda x: cleanData(str(x)))<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Since the fastText tool works with sentences saved in files, we need to create and save a file with all the pre-trained data to be processed:<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_17\"><\/span>In\u00a0[17]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">outfile=open(\"issues_pretrain.txt\", mode=\"w\", encoding=\"utf-8\")\r\nfor line in pretrained.title_desc.values:\r\n    outfile.write(line + '\\n')\r\noutfile.close()<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Finally we can get the pre-trained vectors running the following command:<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p><b>fasttext<\/b>\u00a0skipgram -input\u00a0<i>issues_pretrain.txt<\/i>\u00a0-output\u00a0<i>pretrain_model<\/i>\u00a0-epoch\u00a0<i>100<\/i>\u00a0-wordNgrams\u00a0<i>4<\/i>\u00a0-dim\u00a0<i>300<\/i>\u00a0-minn\u00a0<i>4<\/i>\u00a0-maxn\u00a0<i>6<\/i>\u00a0-lr\u00a0<i>0.01<\/i><\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h3 id=\"Defining-the-training:-an-evaluation-strategy\"><span class=\"ez-toc-section\" id=\"Defining_the_training_an_evaluation_strategy\"><\/span>Defining the training: an evaluation strategy<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Before running the main learning method, let&#8217;s define the strategy that we will use to select the training and evaluation set and also how we are going to evaluate the effectiveness, or how good our model really is.<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h4 id=\"Selecting-Training-and-Testing-sets\"><span class=\"ez-toc-section\" id=\"Selecting_Training_and_Testing_sets\"><\/span>Selecting Training and Testing sets<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">A good model should be able to understand generic hidden patterns in the data and also work well for unseen data. For this reason, we should preserve some examples in our dataset to not be included in the set of examples that will be used for training purposes, so later we can test our model using those examples.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Generally, we would like to have as much representative data as possible for training, and also a large number of different examples to put our model under test. But, sometimes it&#8217;s difficult to have both, mainly when we have small datasets like in our scenario. When this occurs, a technique called\u00a0k-fold cross-validation can help us.<\/p>\n<p>When using this approach, the data is split into k partitions (folds), then in each of the k iterations you select a different folder as the testing set and the rest of the folders are joined to create the new training set. In the end, an average evaluation cryteria of the k iterations is used to measure the quality of our model.<\/p>\n<p>This is a common technique that helps evaluate models to ensure that the results do not depend on how you have selected your training data. K-fold cross-validation is also helpful when your dataset is small (like in our case), because you can ensure that every datapoint will be part of the training and also part of the testing set at any moment. Then, you do not risk leaving out of the training set some important (but few) examples like those that could occur if you use a fixed 80-20 approach for splitting the dataset into a training and testing set.<\/p>\n<p>Although there are some pre-built functionalities in sklearn to work with cross-validation, let\u2019s code our own simple method that given a folder index\u00a0<i>i<\/i>\u00a0(0 &lt;= i &lt; k), it returns a testing set (the dataframe in the folder\u00a0<i>i<\/i>) and a training set that is the union of all the folders different than\u00a0<i>i<\/i>. By doing this you can visualize clearly how it works in the background. Let&#8217;s do it!<\/p>\n<h5><span class=\"ez-toc-section\" id=\"In_37\"><\/span>In [37]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">def rebuild_kfold_sets(folds, k, i):\r\n    training_set = None\r\n    testing_set = None\r\n\r\n    for j in range(k):\r\n        if(i==j):\r\n            testing_set = folds[i]\r\n        elif(training_set is not None):\r\n            training_set = pd.concat([training_set, folds[j]])\r\n        else:\r\n            training_set = folds[j]\r\n    \r\n    return training_set, testing_set<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<h4 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"Defining_the_evaluation_criteria\"><\/span>Defining the evaluation criteria<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h4 id=\"Confusion-matrix,-precision,-recall,-and-F1-score-metrics\"><span class=\"ez-toc-section\" id=\"Confusion_matrix_precision_recall_and_F1_score_metrics\"><\/span>Confusion matrix, precision, recall, and F1 score metrics<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">A common approach when evaluating classification models is to use a confusion matrix. This is a simple but efficient tool to see how well our model is performing at its predictions.<\/div>\n<div class=\"inner_cell\">\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8905\" aria-describedby=\"caption-attachment-8905\" style=\"width: 428px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" class=\"wp-image-8905 size-full\" src=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/conf_matrix.png\" alt=\"\" width=\"428\" height=\"368\" srcset=\"https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/conf_matrix.png 428w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/conf_matrix-300x258.png 300w, https:\/\/uruit.com\/blog\/wp-content\/uploads\/2018\/08\/conf_matrix-20x17.png 20w\" sizes=\"(max-width: 428px) 100vw, 428px\" \/><figcaption id=\"caption-attachment-8905\" class=\"wp-caption-text\">Example of a Confusion matrix<\/figcaption><\/figure>\n<p>As you can see in the image above, the confusion matrix is a square matrix with one row and a column for each class in the classifier. The x-axis is used for the predicted values and the y-axis for the true values. Then a cell <i>(i,j)<\/i>\u00a0in this matrix represents the number of predictions where the class\u00a0<i>j<\/i>\u00a0was predicted but the true class was the class\u00a0<i>i<\/i>. Note that when\u00a0<i>i=j<\/i>\u00a0then\u00a0<i>cell(i,j) = cell(i,i)<\/i>\u00a0which are cells over the diagonal and represent the number of correct predictions that were done for the class\u00a0<i>i<\/i>.<\/p>\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h5><span class=\"ez-toc-section\" id=\"Some_important_metrics_that_we_can_calculate_from_the_values_in_this_matrix_are\"><\/span>Some important metrics that we can calculate from the values in this matrix are:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<ul>\n<li>The overall accuracy (# of correct predictions \/ # total predictions)<\/li>\n<li>Precision for class\u00a0<i>i<\/i>\u00a0(# of correct predictions for class\u00a0<i>i<\/i>\u00a0\/ # total predictions for class\u00a0<i>i<\/i>)<\/li>\n<li>Recall for class\u00a0<i>i<\/i>\u00a0(# of correct predictions for class\u00a0<i>i<\/i>\u00a0\/ # total true items in class\u00a0<i>i<\/i>\u00a0)<\/li>\n<\/ul>\n<p>Also, since it&#8217;s easy to have a high recall with low precision and the opposite (high precision with low recall) it&#8217;s usual to add an F1 score metric (harmonic average of precision and recall) in order to combine precision and recall in just one metric, both being important to increase its value.<\/p>\n<p><i>F1 = 2 x (precision x recall) \/ (precision + recall)<\/i><\/p>\n<p>With this introduction to what a confusion matrix is, and the metrics that we can calculate, now let&#8217;s define a helper method to help us plot a pretty confusion matrix like the one in the image above. Later, after obtaining the final results we will see how to examine a confusion matrix and the metrics in more detail.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_38\"><\/span>In\u00a0[38]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">import itertools\r\n\r\ndef plot_confusion_matrix(cm, classes,\r\n                          normalize=False,\r\n                          title='Confusion matrix',\r\n                          cmap=plt.cm.Blues):\r\n    \"\"\"\r\n    This function prints and plots the confusion matrix.\r\n    Normalization can be applied by setting `normalize=True`.\r\n    \"\"\"\r\n    if normalize:\r\n        cm = cm.astype('float') \/ cm.sum(axis=1)[:, np.newaxis]\r\n\r\n\r\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\r\n    plt.title(title)\r\n    plt.colorbar()\r\n    tick_marks = np.arange(len(classes))\r\n    plt.xticks(tick_marks, classes, rotation=45)\r\n    plt.yticks(tick_marks, classes)\r\n\r\n    fmt = '.2f' if normalize else 'd'\r\n    thresh = cm.max() \/ 2.\r\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\r\n        plt.text(j, i, format(cm[i, j], fmt),\r\n                 horizontalalignment=\"center\",\r\n                 color=\"white\" if cm[i, j] &gt; thresh else \"black\")\r\n\r\n    plt.tight_layout()\r\n    plt.ylabel('True label')\r\n    plt.xlabel('Predicted label')\r\n    \r\ndef plot_confusion_matrix_with_accuracy(classes, y_true, y_pred, title, sum_overall_accuracy, total_predictions):\r\n    cm = ConfusionMatrix(y_true, y_pred) \r\n    print('Current Overall accuracy: ' + str(cm.stats()['overall']['Accuracy']))\r\n    if total_predictions != 0:\r\n        print('Total Overall Accuracy: ' + str(sum_overall_accuracy\/total_predictions))\r\n    else:\r\n        print('Total Overall Accuracy: ' + str(cm.stats()['overall']['Accuracy']))\r\n\r\n    conf_matrix = confusion_matrix(y_true, y_pred)\r\n    plt.figure()\r\n    plot_confusion_matrix(conf_matrix, classes=classes, title=title)\r\n    plt.show()<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h3 id=\"Main-method-(k-fold-cross-validation,-with-oversampling-in-training-folders)\"><span class=\"ez-toc-section\" id=\"Main_method_k-fold_cross-validation_with_oversampling_in_training_folders\"><\/span>Main method (k-fold cross-validation, with oversampling in training folders)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Now that we have our data ready, a python wrapper for the FastText classifier and a strategy for getting the training and testing sets, we can finally present the main learning method.<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<h5 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"In_39\"><\/span>In\u00a0[39]:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">from sklearn.metrics import confusion_matrix\r\nfrom pandas_ml import ConfusionMatrix\r\n\r\n# K-folds cross validation \r\n# K=5 or K=10 are generally used. \r\n# Note that the overall execution time increases linearly with k\r\nk = 5\r\n\r\n# Define the classes for the classifier\r\nclasses = ['0','1','2']\r\n\r\n# Make Dataset random before start\r\ndf_rand = df.sample(df.storypoint.count(), random_state=99)\r\n\r\n# Number of examples in each fold\r\nfsamples =  int(df_rand.storypoint.count() \/ k)\r\n\r\n# Fill folds (obs: last folder could contain less than fsamples datapoints)\r\nfolds = list()\r\nfor i in range(k):\r\n    folds.append(df_rand.iloc[i * fsamples : (i + 1) * fsamples])\r\n        \r\n# Init\r\nsum_overall_accuracy = 0\r\ntotal_predictions = 0\r\n\r\n# Repeat k times and average results\r\nfor i in range(k):\r\n    \r\n    #1 - Build new training and testing set for iteration i\r\n    training_set, testing_set  = rebuild_kfold_sets(folds, k, i)\r\n    y_true = testing_set.storypoint.tolist()\r\n\r\n    #2 - Oversample (ONLY TRAINING DATA)\r\n    X_resampled, y_resampled =  SimpleOverSample(training_set.label_title_desc.values.tolist(), training_set.storypoint.values.tolist())\r\n    \r\n    #3 - train\r\n    clf = FastTextClassifier()\r\n    clf.fit(X_resampled, y_resampled)\r\n    \r\n    #4 - Predict\r\n    y_pred = clf.predict(testing_set.label_title_desc.values.tolist())\r\n         \r\n    #3 - Update Overall Accuracy\r\n    for num_pred in range(len(y_pred)):\r\n        if(y_pred[num_pred] == y_true[num_pred]):\r\n            sum_overall_accuracy += 1\r\n        total_predictions += 1\r\n\r\n    #4 - Plot Confusion Matrix and accuracy \r\n    plot_confusion_matrix_with_accuracy(classes, y_true, y_pred, 'Confusion matrix (testing-set folder = ' + str(i) + ')', sum_overall_accuracy, total_predictions)<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre class=\"\">Current Overall accuracy: 0.6660869565217391\r\nTotal Overall Accuracy: 0.6660869565217391\r\n<\/pre>\n<\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAIABJREFUeJzt3XmcFNW5\/\/HPF8a4ASKCBhHEBXFLVEQ0atz361XUGBfiEhc0V5O4XzXmpzESN9RoMDF6NYoLxLgkxn3BuEVQQVwILriCIIIigiyB8fn9UTXaTma6a7Cb6p75vnnVi+6q6lNP91Q\/fc6pqlOKCMzMrLh2eQdgZlYLnCzNzDJwsjQzy8DJ0swsAydLM7MMnCzNzDJodclS0vKS\/i5ptqS\/fINyBkl6uJyx5UXS9yW9\/g1ef6Gkk8oZU5FtzZW09tLYVrm0ZJ+T1FtSSKprZvl5km6pTKTlIelnki7KO46lLbdkKelQSS+kX45pkh6QtG0Ziv4BsBqwSkQcuKSFRMStEbFbGeKpqPSLt26xdSLiqYjou4TldwMOB\/6YPt9B0pQlKauJsv8h6ZjCeRHRISLeLkf55ZDx\/ZZln6smaVJ\/XNI8Sa9J2qVg8bXAjyStmld8ecglWUo6Bfgt8BuSnawX8Htg3zIUvybwRkQsLkNZNa+5GkwLHAncHxHzyxBOa5X7PqdEOb\/PI4AXgVWAXwB3pD+cRMQC4AGSH9G2IyKW6gSsBMwFDiyyzrIkyXRqOv0WWDZdtgMwBTgV+AiYBvw4XfYr4N\/AonQbRwPnAbcUlN0bCKAufX4k8DYwB3gHGFQw\/+mC120NPA\/MTv\/fumDZP4BfA8+k5TwMdG3mvTXEf0ZB\/AOBvYA3gE+AswvWHwA8C3yarjsM+Fa67Mn0vXyevt+DCsr\/X+BD4OaGeelr1km30S99vjowE9ihmXhHAT9KH68IzAe+SLc3N319O+BM4C3gY+B2oEv6muWAW9L5n6af3WrAEKAeWJCWMyxdP4B108c3AlcD96Wf6xhgnYLYdgNeT\/8mvweeAI5p5n0IuCL9zGcDLwMbF+xvQ4H3genANcDyzb3fRuU2tc+1A84B3ku3NxxYqZn9b6007jnAI+nft3B\/3Qr4Z\/rZvVT4dyLZ74aQ7HfzGz63MnxH1wMWAh0L5j0FHF\/wfBDw+NLOH3lOS3+DsAewuGFnaWad84HRwKpAt3Rn+XW6bIf09ecDy5AkmXnAyuny8xrtbI2ff7mzpl+Gz4C+6bLuwEbp4yNJkyXQBZgFHJa+7pD0+SoFO+1b6U62fPr8ombeW0P8\/y+N\/1hgBnAb0BHYiCSBrJ2uv3n6halLY58InFRQ3pfJpVH5F5MkgeUpSJbpOsem5awAPAQMLfK3mAFs0aj8KY3WOSn9e62RbvOPwIh02XHA39NttU\/fT6eCz+2YRmU1TpafkPxg1AG3AiPTZV3Tv93+6bKfkySs5pLl7sBYoDNJ4twA6J4u+y1wT\/p37pjGe2Fz77eJss\/j6\/vYUcAkYG2gA3AXcHPj\/S99\/ixwefq5bUeSNG9Jl\/Ug+ZHZiyQB75o+71bw+b2f7jN1wDJNxHYvSaJtarq3mfezHzCx0bxhwO8KnvcDPlna+SPPKY9m+CrAzCjeZBkEnB8RH0XEDJJf78MKli9Kly+KiPtJftGXqE+OpNawsaTlI2JaRExoYp3\/At6MiJsjYnFEjABeA\/67YJ0\/RcQbkTRXbwc2LbLNRcCQiFgEjCT54l8ZEXPS7U8AvgsQEWMjYnS63XdJEtH2Gd7TuRGxMJpoPkfEdcCbJDW17iTNrOZ0JvkCF3Mc8IuImBIRC0mSxw\/SLoBFJH\/zdSOiPn0\/n5Uor9BdEfFcur\/cylef617AhIi4K112FUlNujmLSBLh+oAiYmJETJMkkh+PkyPik4iYQ9I9dHALYmxsEHB5RLwdEXOBs4CDG3eJSOoFbAH8Mv1bPUmSqBv8iKQL5P6I+CIiHgFeSN97gxsjYkK6fyxqHEhE7B0RnZuZ9m4m\/g4kte9Cs0k+vwZzSFqJbUYeyfJjoGuJvrTVSZowDd5L531ZRqNkO4\/kD9wiEfE5SdP1eGCapPskrZ8hnoaYehQ8L\/yilorn44ioTx83JLPpBcvnN7xe0nqS7pX0oaTPSL7IXYuUDTAjkn6lYq4DNiapLSwsst4svv4lacqawN2SPpX0KUmttZ6kuX0zSe11pKSpki6RtEyJ8go197muDkxuWBBJdefLAzGSJqQHD+dK+n5EjCKpHV0NTJd0raROJC2XFYCxBfE\/mM5fUk3tv3Ukn0fj9Wal+2Hhug3WBA5siCuNbVuSH7gGkym\/uUCnRvM68fUfzY78Z0Jt1fJIls+SNDMHFllnKsmO0qBXOm9JfE7yZWjw7cKFEfFQROxKsgO+RpJESsXTENMHSxhTS\/yBJK4+EdEJOJukGVlM0aGkJHUgaXpeD5wnqUuR1V8m6V4oVvZkYM9GtZblIuKDtPb\/q4jYkKTfd2++OjDwTYa8mkbS7G94Typ8HhEbRXJkvUNEPJXOuyoiNidptq4HnE7SXzufpPulIfaVIqIhKS9JjE3tv4v5+g9iw3tYWdKKjdZtMJmk+V74ua4YEYWn7ZT6Wz9Q8KPReHqgmZdNANaWVPgjuUk6v8EGJH2obcZST5YRMZukv+5qSQMlrSBpGUl7SrokXW0EcI6kbpK6pusv6bln44HtJPWStBJJkwgASatJ2ifdWReS\/KLWN1HG\/cB66elOdZIOAjYk6Q+qtI4kfXNz01rvTxotn07SN9YSVwJjI+IYkoMn1xRZ936+3uyfDqySfpYNrgGGSFoTktONJO2bPt5R0ncktU\/fxyK++oyXJPYG9wHfSfehOuAEGv0QFpK0haQt01rt5yQ\/2PUR8QXJD+QVDafCSOohafci77eUEcDJktZKf5h+A\/y5cddTRLxH0qz+laRvpafOFXbt3AL8t6TdJbWXtFx6KtMaZBQRexb8aDSe9mzmNW+QfG\/OTbe5H0m30J0Fq21PckS8zcjl1KGIuBw4heSI4QySX9ATgb+mq1xAshO9DLwCjEvnLcm2HgH+nJY1lq8nuHYkR9WnkhxI2B74nybK+JikRnQqSTfCGcDeETFzSWJqodOAQ0maQNeRvJdC5wE3pc20H5YqLE1ie5B0PUDyd+gnaVAzLxkO7CVpeYCIeI0kGbydbnN1kuR7D\/CwpDkkB3u2TF\/\/beAOkkQ5keTIb8MP35UkfZuzJF1VKvZC6Wd\/IHAJyd9kQ5J9prkuhU4kn98skqbuxyRHwCE5c2ASMDrt6niUtA+8mfdbyg0k3Q9PkpxhsQD4aTPrHkryWX0CnEvyeTe8x8kkp9OdzVffk9NZOt\/bg4H+JJ\/XRcAP0uMHSFqOpN\/0pqUQR9VQ0tVj1jxJvwE+iojf5h1Lc9JzDKeQnPr1eN7xtGaSfgr0jIgz8o5laXKytJqVNpXHkPQ5nk7SFF+7qTMAzL6pVndtuLUp3yM5v3UmSV\/fQCdKqxTXLM3MMnDN0swsg286yEJZrdK1a\/Tq1TvvMGpK\/RduGbRUXftSp6laY++\/9y4zZ84s6wfXvtOaEYuz95rE\/BkPRcQe5YyhJaoqWfbq1ZvHnxmTdxg15bP5Hlyppbqs2JILiAxgu60HlL3MWDyfZfuWPNvtSwvGX13qyrWKqqpkaWZtiaCso8pVlpOlmeVDgGqnS8TJ0szy45qlmVkpgnbt8w4iMydLM8uPm+FmZiUIN8PNzEqTa5ZmZpm4ZmlmloFrlmZmpfikdDOz0nxSuplZRq5ZmpmVImjvk9LNzIrzeZZmZhm5z9LMrBQfDTczy8Y1SzOzDFyzNDMrQb423MwsG9cszcwyqKGaZe2kdTNrZdKj4VmnYiVJPSU9LmmipAmSfp7OP0\/SB5LGp9NeBa85S9IkSa9L2r1UtK5Zmlk+RDlvK7EYODUixknqCIyV9Ei67IqIGPq1TUsbAgcDGwGrA49KWi8i6pvbgGuWZpaT8tUsI2JaRIxLH88BJgI9irxkX2BkRCyMiHeASUDRm6M7WZpZfhqOiGeZoKukFwqmwU0Xqd7AZsCYdNaJkl6WdIOkldN5PYDJBS+bQvHk6ma4meWoZUfDZ0ZE\/6LFSR2AO4GTIuIzSX8Afg1E+v9lwFEknQCNRbGynSzNLD9lPBouaRmSRHlrRNwFEBHTC5ZfB9ybPp0C9Cx4+RrA1GLluxluZvlQWY+GC7gemBgRlxfM716w2n7Aq+nje4CDJS0raS2gD\/BcsW24Zmlm+SlfzXIb4DDgFUnj03lnA4dI2pSkif0ucBxAREyQdDvwL5Ij6ScUOxIOTpZmliOVKVlGxNM03Q95f5HXDAGGZN2Gm+GNnHjcMfRZszvf67\/Jl\/N+efYZDNh0I7YZsBk\/OugAZn\/6aY4RVp8FCxawzy7bssd2W7DL1ptx+UXnA\/Cz445gxwHfYddt+nHaTwezaNGinCOtXsOu+i1bbPYdBvT7Lj8+7FAWLFiQd0gVl9yCR5mnvDlZNnLIYYdzx1\/v+9q8HXfahX++8BLPPPci6\/Tpw+VDL8opuuq07LLLMuKvD\/Lgk8\/zwBPP8cRjjzDu+TEM\/MEhjBrzMg8\/PZaFC+Yz8uY\/5R1qVZr6wQdcc\/XvePKfz\/HcuJep\/6KeO24fmXdYlSehdtmnvDlZNrLNttuxcpcuX5u30y67UVeX9FhsscVWTP3ggzxCq1qSWLFDBwAWL1rEosWLkMROu+7xZa1gk35bMG3qlJwjrV6LFy9m\/vz5LF68mHnz5tG9++p5h7RUuGbZit0y\/E\/sstseeYdRderr69lz+wH0W78n399+Zzbr\/9XFEIsWLeKu229jh513yzHC6rV6jx787ORT2bBPb9bt3YOVOq3Ezru2jc\/KyTIlaY\/0IvVJks6s5LaWhqEX\/4a6ujp+ePCheYdSddq3b88DTzzH6FfeYvyLz\/P6xAlfLjvn9J+x5fe2ZcD3ts0xwuo1a9Ys7vv7Pbzy2lu8+c4UPp\/3OSNvuyXvsJYKJ0tAUnvgamBPYEOSQ\/gbVmp7lTbiluE8\/MB9XPunm6viD1etVlqpM9\/bZjv+8djDAPz2kgv4ZOZMfnnBJTlHVr3+MepR1uzdm27durHMMsuwz777MWb0s3mHVXlq4ZSzStYsBwCTIuLtiPg3MJLk4vWa8+jDD3Ll5Zdy21\/+ygorrJB3OFXn45kzmD07OUNgwfz5PP3EKNbt05cRN9\/AE6Me5XfXDaddO\/f4NGeNnr14\/rkxzJs3j4jgH4+Pou\/6G+QdVsWJ7LXKaqigVPI8y6YuVN+y8UrpxfCDIdlp8nb0EYN45skn+PjjmWy07pqcec65XDH0YhYuXMh+eyd9lf0HbMkVv\/t9zpFWj4+mf8gpJxzDF\/X1fPHFF+w98AB23n0v1l51RXr07MV+e2wPwB5778vPT\/9FztFWny0GbMnA\/Q5g2636U1dXxyabbMqPjz4277CWimpIglkpoui140tesHQgsHtEHJM+PwwYEBE\/be41m\/XrH48\/M6a5xdaEz+YvzjuEmtNlxWXyDqHmbLf1AMaNfaGsma1ulbWj014XZF5\/1i2DxpYaSKOSKlmzbPGF6mbWttRSzbKSHUnPA30krSXpWySjEt9Twe2ZWS2psQM8FatZRsRiSScCDwHtgRsiYkKJl5lZGyFUUwf+KjqQRkTcT5EL2c2sbaulZrhHHTKz\/NROrnSyNLOcyDVLM7NMnCzNzDJwsjQzK6Hhcsda4WRpZvmpnVzpZGlmOfEBHjOzbJwszcwyqIZ762TlZGlmuXHN0syshGoZ1DcrJ0szy42TpZlZBk6WZmZZ1E6udLI0s\/y4ZmlmVopPSjczK01ADeVKJ0szy4to55PSzcxKczPczKwU1VYzvHZurWZmrYqAdu2UeSpaltRT0uOSJkqaIOnn6fwukh6R9Gb6\/8rpfEm6StIkSS9L6lcqXidLM8uNlH0qYTFwakRsAGwFnCBpQ+BM4LGI6AM8lj4H2BPok06DgT+U2oCTpZnlpuH68CxTMRExLSLGpY\/nABOBHsC+wE3pajcBA9PH+wLDIzEa6Cype7FtOFmaWT5aUKtMc2VXSS8UTIObLFbqDWwGjAFWi4hpkCRUYNV0tR7A5IKXTUnnNcsHeMwsF8l5li06wjMzIvoXLVPqANwJnBQRnxUpv6kFUaxsJ0szy0l5h2iTtAxJorw1Iu5KZ0+X1D0ipqXN7I\/S+VOAngUvXwOYWqx8N8PNLDflOsCjJOteD0yMiMsLFt0DHJE+PgL4W8H8w9Oj4lsBsxua681xzdLM8iHKeQXPNsBhwCuSxqfzzgYuAm6XdDTwPnBguux+YC9gEjAP+HGpDThZmlkulqDPslkR8TTND\/i2cxPrB3BCS7bhZGlmuamlK3icLM0sN7423MwsgxrKldWVLCVYts4H6Fui786n5h1CzRl338V5h1Bz\/r34i\/IX6sF\/zcxK8+C\/ZmaZ+L7hZmaZ1FCudLI0s5yU96T0inOyNLNclPOk9KXBydLMcuNkaWaWQQ3lSidLM8uPa5ZmZqXU2N0dnSzNLBfyeZZmZtnUUK50sjSz\/LSroWzpZGlmuamhXOlkaWb5kKC9r+AxMyutVRzgkdSp2Asj4rPyh2NmbUkN5cqiNcsJJDcdL3w7Dc8D6FXBuMyslRPJ6UO1otlkGRE9m1tmZlYONdRlSaZ7OEg6WNLZ6eM1JG1e2bDMrNVTclJ61ilvJZOlpGHAjiQ3MIfkhuTXVDIoM2sbpOxT3rIcDd86IvpJehEgIj6R9K0Kx2VmrZxofSelL5LUjuSgDpJWASpwqzcza2tqKFdm6rO8GrgT6CbpV8DTgO8lambfWC31WZasWUbEcEljgV3SWQdGxKuVDcvMWrvWegVPe2ARSVM80xF0M7NSaidVZjsa\/gtgBLA6sAZwm6SzKh2YmbV+raoZDvwI2Dwi5gFIGgKMBS6sZGBm1rolR8PzjiK7LMnyvUbr1QFvVyYcM2szqqTGmFWzzXBJV0i6nOQk9AmS\/k\/SdcArwKdLK0Aza73KeVK6pBskfSTp1YJ550n6QNL4dNqrYNlZkiZJel3S7qXKL1azbNjgBOC+gvmjS4dtZlZamWuWNwLDgOGN5l8REUMbbXdD4GBgI5LjMY9KWi8i6psrvNhAGtcvacRmZqWUu88yIp6U1Dvj6vsCIyNiIfCOpEnAAODZ5l6Q5Wj4OpJGSnpZ0hsNU8aAWoX6+nq22qIf+w\/877xDqRprrNaZB6\/9GS\/eeQ5j7\/gFJxyyAwDfXa8HT9x0KqNHnsnTt55B\/43WBGC93qvxj5tO5dMxV3DSYTvnGHl1qa+vZ\/\/dtuYnh\/8AgGefepwDdt+G\/Xb9Hj8auCvvvfNWzhFWVguPhneV9ELBNDjjZk5M89cNklZO5\/UAJhesMyWd16ws50zeCPyJ5IdgT+B2YGTGIFuFq393Jeuvv0HeYVSVxfVfcObld7HZARew\/eFDOe6g7Vh\/7W8z5KSBDLn2AbY6+CJ+\/Yd7GXLSQABmzf6cUy\/+C78dPirnyKvLzf\/3e9bp0\/fL5+efdTKXDLueux95lv8aeCB\/vPKSHKOrLAnaS5knYGZE9C+Yrs2wmT8A6wCbAtOAyxo238S6UaygLMlyhYh4CCAi3oqIc0hGIWoTpkyZwoMP3M+RRx2ddyhV5cOZnzH+tSkAzJ23kNfe+ZDVu3UmAjqtuBwAK3VYnmkzZgMwY9Zcxv7rfRYtbrZLqM35cOoHPPHYgxxwyBFfzpPE3DlzAJgz5zO6rdY9r\/CWikqPOhQR0yOiPiK+AK4jaWpDUpMsHLN3DWBqsbKynDq0UEkd+C1JxwMfAKu2POzadMapJ3PBhRd\/uQPbf+rVvQub9l2D5199l9OH3sHfrz6BC0\/ej3btxI5HXla6gDbqonPP4LRzLuDzuV\/tW+cPHcbxhx3Acsstx4odOzLy74\/nGGHlVfrUIUndI2Ja+nQ\/vjpwfQ\/JBTaXkxzg6QM8V6ysLDXLk4EOwM+AbYBjgaMyBPkfh\/Frzf333Uu3VbvRr5\/HOm7Oist\/ixFDj+H0oXcy5\/MFDD7w+5xx2V302fOXnDH0Tv5w7qC8Q6xK\/3jkAbp07cZG393sa\/OHXzeMa26+k8fHvsF+Bx3Gxb9q3RfLlfnUoREkB2j6Spoi6WjgEkmvSHqZpEV8MkBETCDpUvwX8CBwQrEj4ZBtII0x6cM5fDUAcBY30vRh\/Jox+p\/PcN+9f+ehBx9gwYIFzPnsM4464jBuuOnmvEOrCnV17Rgx9Fj+\/MAL\/G3USwAM2ntLTr3kDgDufORFfv\/\/Ds0zxKo17oXRPP7w\/Tw56mEWLlzA53PmcPxhB\/DOW2+wSb8tANhznwMYPGhgzpFWjlBZx7OMiEOamN3sWT0RMQQYkrX8Yiel3y3pruamUgVHxJPAJ1kDqUbnD7mQSe9M5rU332H4LSPYfsednCgLXHPuIF5\/50OuuuWrgzbTZszm+5v3AWCHAesx6f0ZeYVX1U4561c8PvYNHh3zLy77\/Y1suc32DPvTn5nz2WzefetNAJ59ctTXDv60Oi2oVVbDhT7FapbDlkYA6eH\/wQA9e\/mGkbVi603XZtDeW\/LKGx8weuSZAJw77B5O+PVtXHr6D6ira8fChYs58YIRAKy2SkeeufUMOq64HF9EcOKgHdjsgCHM+XxBnm+jqtTV1XH+pcP4+eBBtFM7OnXuzAWX\/SHvsCqqli53VETRo+XfrPDkBNF7I2LjLOv327x\/PDP6+YrF0xp1GfDTvEOoOePu89jVLXXgnt\/n1ZfGlTWzrbruxnHQpX\/JvP6w\/TccGxH9yxlDS2Qdz9LMrKxEbdUsnSzNLDe1NERb5lHPJS3bkoKbOYxvZgZ8dVuJrFPeStYsJQ0gOfy+EtBL0ibAMRFRtLOsmcP4ZmZfqoIcmFmWmuVVwN7AxwAR8RJt6HJHM6uc1nLqUIN2EfFeo45YX+BrZt9IMkRbFWTBjLIky8lpUzwktQd+CrSpIdrMrDJq6VaxWZLlT0ia4r2A6cCj6Twzs2+khiqWma4N\/4hk+HUzs7KRyntteKVlORp+HU0MihkRWUcpNjNrUg3lykzN8EcLHi9HMibc5GbWNTPLrJZOHcrSDP9z4XNJNwOPVCwiM2sTBFVxsnlWS3K541rAmuUOxMzaGLWymqWkWXzVZ9mOZIzKMysZlJm1DWryvmHVqWiyTO+9swnJfXcAvohKjulmZm1Gue8bXmlFzwlNE+Pd6d3R6p0ozayc2in7lLcsJ9A\/J6lfxSMxszZHUuYpb802wyXVRcRiYFvgWElvAZ+T1J4jIpxAzWyJ1VozvFif5XNAP6D13l7OzPJTJaMJZVUsWQogIt5aSrGYWRvTWi537CbplOYWRsTlFYjHzNqI1tQMbw90gBo6EcrMaoho30pqltMi4vylFomZtSnJ3R3zjiK7kn2WZmYVUSXnT2ZVLFnuvNSiMLM2qVUc4ImIT5ZmIGbWtrSmZriZWUW1ipqlmVml1VCudLI0s3yI1nd3RzOz8hNVMUBGVrWU2M2slVELppJlSTdI+kjSqwXzukh6RNKb6f8rp\/Ml6SpJkyS9nGVkNSdLM8uFgPZS5imDG4E9Gs07E3gsIvoAj\/HVXR72BPqk02DgD6UKd7I0s9xI2adSIuJJktveFNoXuCl9fBNfjaK2LzA8EqOBzpK6FyvffZZmlpMWD+rbVdILBc+vjYhrS7xmtYiYBhAR0yStms7vwddv6T0lnTetuYKcLM0sF0twNHxmRPQv4+YbK3rbHCdLM8vNUjgaPl1S97RW2R34KJ0\/BehZsN4awNRiBbnP0sxyU86j4c24BzgifXwE8LeC+YenR8W3AmY3NNebU1U1y+Ra0do576oa\/OmGs\/IOoea8+fGcvEOoOQsW15e\/0DKfZylpBLADSd\/mFOBc4CLgdklHA+8DB6ar3w\/sBUwC5gE\/LlV+VSVLM2s7yn0FT0Qc0syi\/xhBLb2t9wktKd\/J0sxyU0stSSdLM8tNaxn818ysYpJmeO1kSydLM8tNDbXCnSzNLC9CrlmamZXmmqWZWQnuszQzyyLjaELVwsnSzHLjZGlmloEP8JiZlSB8UrqZWSa+b7iZWQZuhpuZleBmuJlZJr6Cx8ysNJ9naWaWTQ3lSidLM8tH0mdZO+nSydLMclM7qdLJ0szyVEPZ0snSzHLjZriZWQa1kyqdLM0sTzWULZ0szSwXwpc7mpmV5pPSzcyyqaFc6WRpZjmqoWzpZGlmOfFAGmZmmbjPspWYPHkyx\/z4cKZP\/5B27dpx1NGDOfFnP887rKr0wG3\/x6i7RxAR7LTfoew16BjefX0C1w85k0X\/Xki79nUcddYQ1t14s7xDrRp\/v+U6Hr7zVoJgt\/0Hsc9hg5kzexaXnn48H02dzKqr9+SMoX+kQ6fOeYdaEaKmWuG0yzuAalZXV8dFl1zG+Fcm8sTTo\/njNVcz8V\/\/yjusqjN50muMunsEFwy\/l4tHPsyLTz3KtPff5rYrh3DAcSdz0ciHOfAnp3LblUPyDrVqvPfmazx8560Mve1+rvzLYzz\/5KNMfe9t7rx+GN\/dcluuufeffHfLbbnz+mF5h1pRkjJPeXOyLKJ79+5s1q8fAB07dmT99Tdg6tQPco6q+nzwziT6fGczll1+edrX1bHB5lvx\/KgHEWL+3LkAzJs7h5W7rZZzpNVjyjtvst53N2fZ5VegfV0dG\/ffitGPPcCYxx9ip31+CMBO+\/yQ0aMezDnSypKyT3lzsszovXffZfz4F9liwJZ5h1J1eq7Tl4njxjDn01ksnD+f8U+P4uPpUzn8tPO49coLOGHPLbj1il9z8Iln5R1q1ei1bl\/+NW40n336CQvnz2PsU6OYOX0qsz+ZQZf0R6VLt9WY\/cnMnCOtLLVgKlmW9K6kVySNl\/RCOq+LpEckvZn+v\/KSxlqxPktJPYHhwLeBL4BrI+LKSm2vkubOncshPzyASy\/7LZ06dco7nKrTY+0+7HPk\/\/Cb\/zmE5ZZfkV7rbUj79nU8csdwDjv1XLbc+b949uG\/c+35p\/GLa0bmHW5V6Ln2euz\/4xM4d\/BBLLfCivTuuyHt2rfPO6ylqzKdljskppDqAAAJUklEQVRGROEvzJnAYxFxkaQz0+f\/uyQFV7JmuRg4NSI2ALYCTpC0YQW3VxGLFi3ikB8ewEGHDGLgfvvnHU7V2nHgIVx424Oce\/2ddOjUmW\/3Wosn772DATvtBcBWu+7NWxPG5xxlddl1\/0O54vZHuPDGv9KxU2dW77U2K3XpxiczpgPwyYzprNSla85RVpZa8G8J7QvclD6+CRi4pAVVLFlGxLSIGJc+ngNMBHpUanuVEBEcf+zR9F1\/A35+8il5h1PVGpqLM6d9wPOPP8DWe+zLyl1XY+LYZwGY8NwzfLvnWnmGWHU+\/Tj5zGZMm8Kzj93PdnsNZMAOuzHqntsBGHXP7Wy54+55hlhRosV9ll0lvVAwDW5UZAAPSxpbsGy1iJgGSU4CVl3SeJfKqUOSegObAWOWxvbK5Z\/PPMNtt97Mxht\/hy033xSAX13wG\/bYc6+cI6s+V5w2mLmzZ9G+ro4f\/+8QOnTqzLG\/vIThl55Lff1illl2WY455+K8w6wqF59yNJ\/NnkVd3TIcd\/aFdOjUmQOOPpFLTzuOR+8eQbdv9+CMy67NO8yKamF9cWZE9C+yfJuImCppVeARSa99k9gaU0SUs7z\/3IDUAXgCGBIRdzWxfDAwGKBnr16bv\/HWexWNp7X56ys+Ot9SK9S1sb7BMjjl4N2ZNOGlsvYwbrxJv\/jLg09lXn\/D1TuMLZEsvyTpPGAucCywQ0RMk9Qd+EdE9F2SeCt6NFzSMsCdwK1NJUqAiLg2IvpHRP9uXbtVMhwzqzLl6rOUtKKkjg2Pgd2AV4F7gCPS1Y4A\/raksVbyaLiA64GJEXF5pbZjZrWrXfnqqqsBd6cnr9cBt0XEg5KeB26XdDTwPnDgkm6gkn2W2wCHAa9IajgMenZE3F\/BbZpZLSlTsoyIt4FNmpj\/MbBzObZRsWQZEU9TW5d+mtlS5JHSzcyyqJLLGLNysjSz3NRQrnSyNLMc1VC2dLI0s5x4pHQzs0zcZ2lmVkKtjZTuZGlm+amhbOlkaWa5aVdD7XAnSzPLTe2kSidLM8uLT0o3M8uqdrKlk6WZ5aJhpPRa4WRpZrmpoVzpZGlm+XHN0swsA1\/uaGaWRe3kSidLM8tPDeVKJ0szy4fkK3jMzLKpnVzpZGlm+amhXOlkaWb5qaFWuJOlmeXFI6WbmZVUa5c7tss7ADOzWuCapZnlppZqlk6WZpYb91mamZWQnJSedxTZOVmaWX6cLM3MSnMz3MwsAx\/gMTPLoIZypZOlmeWohrKlk6WZ5aaW+iwVEXnH8CVJM4D38o6jCV2BmXkHUWP8mS2Zav3c1oyIbuUsUNKDJO83q5kRsUc5Y2iJqkqW1UrSCxHRP+84aok\/syXjz616+dpwM7MMnCzNzDJwsszm2rwDqEH+zJaMP7cq5T5LM7MMXLM0M8vAydLMLAMnSzOzDJwsmyGpr6TvSVpGUvu846kV\/qxaRtK6kvpLWjbvWKw4H+BpgqT9gd8AH6TTC8CNEfFZroFVMUnrRcQb6eP2EVGfd0zVTtLeJPvZx8CHwLkNn6FVH9csG5G0DHAQcHRE7Az8DegJnCGpU67BVan0Sz9e0m0AEVHvGmZxkrYGhgJHRMSOwCzgzHyjsmKcLJvWCeiTPr4buBf4FnCoVEsj8FWepBWBE4GTgH9LugWcMDO6KCJeTB+fC3Rxc7x6OVk2EhGLgMuB\/SV9PyK+AJ4GxgPb5hpcFYqIz4GjgNuA04DlChNmnrFVuTHAXfBlP++ywJokP9RIWiW\/0KwpTpZNewp4GDhM0nYRUR8RtwGrA5vkG1r1iYipETE3ImYCxwHLNyRMSf0krZ9vhNUn3aca+sAFfAp8EhEzJA0CLpC0fH4RWmMez7IJEbFA0q1AAGelX\/aFwGrAtFyDq3IR8bGk44BLJb0GtAd2zDmsqhYRi4G5kiZLuhDYDTgyIubnHJoVcLJsRkTMknQd8C+S2tIC4EcRMT3fyKpfRMyU9DKwJ7BrREzJO6ZqlvaDLwN8P\/1\/54h4M9+orDGfOpRB2qcUaf+llSBpZeB24NSIeDnveGqFpCOB5yNiQt6x2H9ysrSKkLRcRCzIO45aIknhL2TVcrI0M8vAR8PNzDJwsjQzy8DJ0swsAydLM7MMnCxbCUn1ksZLelXSXySt8A3K2kHSvenjfSQ1O8CDpM6S\/mcJtnGepNOyzm+0zo2SftCCbfWW9GpLYzQr5GTZesyPiE0jYmPg38DxhQuVaPHfOyLuiYiLiqzSGWhxsjSrNU6WrdNTwLppjWqipN8D44CeknaT9KykcWkNtAOApD0kvSbpaWD\/hoIkHSlpWPp4NUl3S3opnbYGLgLWSWu1l6brnS7peUkvS\/pVQVm\/kPS6pEeBvqXehKRj03JeknRno9ryLpKekvRGOkQcktpLurRg28d90w\/SrIGTZSsjqY7kMsNX0ll9geERsRnwOXAOsEtE9CMZ1PgUScsB1wH\/TXLJ3bebKf4q4ImI2AToB0wgGYPxrbRWe7qk3UiGtxsAbApsLmk7SZsDBwObkSTjLTK8nbsiYot0exOBowuW9Qa2B\/4LuCZ9D0cDsyNii7T8YyWtlWE7ZiX52vDWY3lJ49PHTwHXk4yS9F5EjE7nbwVsCDyTDsv5LeBZYH3gnYbrkdMRgwY3sY2dgMPhy+HXZqeXNhbaLZ0axmnsQJI8OwJ3R8S8dBv3ZHhPG0u6gKSp3wF4qGDZ7enlp29Kejt9D7sB3y3oz1wp3bZHH7dvzMmy9ZgfEZsWzkgT4ueFs4BHIuKQRuttSjLCUjkIuDAi\/thoGyctwTZuBAZGxEvpddM7FCxrXFak2\/5pRBQmVST1buF2zf6Dm+Fty2hgG0nrAkhaQdJ6wGvAWpLWSdc7pJnXPwb8JH1t+\/Q2G3NIao0NHgKOKugL7SFpVeBJYD9Jy0vqSNLkL6UjME3JrT4GNVp2oKR2acxrA6+n2\/5Juj6S1lMykrvZN+aaZRuSDix7JDBCX92+4JyIeEPSYOA+STNJRobfuIkifg5cK+looB74SUQ8K+mZ9NScB9J+yw2AZ9Oa7VySoe3GSfozyYjz75F0FZTyS5IRxd8j6YMtTMqvA0+QjDF6fDoG6f+R9GWOS4c9mwEMzPbpmBXngTTMzDJwM9zMLAMnSzOzDJwszcwycLI0M8vAydLMLAMnSzOzDJwszcwy+P+Acwzkk1RwEQAAAABJRU5ErkJggg==\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre class=\"\">Current Overall accuracy: 0.6852173913043478\r\nTotal Overall Accuracy: 0.6756521739130434<\/pre>\n<\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAIABJREFUeJzt3XmcFNXZ9vHfBQOIAqKCKAOIC2jQRNyNGqPRuEVFk\/i4EKNxQX01CYnRaDY1ajQJ0cQlMWp81KC4RPO4R3Ff4ga4ElwQRTYVXJFFYLzfP6oG28lMdw10U90z15dPf+iuqj51d0\/13eecOnVaEYGZmRXXIe8AzMxqgZOlmVkGTpZmZhk4WZqZZeBkaWaWgZOlmVkGbS5ZSuoq6TZJH0q6cTnKGS7pnnLGlhdJX5H08nI8\/xxJI8sZU5F9fSxpvRWxr3JpzTEnaaCkkFTXwvrTJY2uTKTlIWlfSdflHceKlluylHSIpHHph2OWpLsk7VCGor8N9AHWiIgDlrWQiLgmInYrQzwVlX7wNii2TUQ8EhEbLmP5vYHvAn9NH+8kafqylNVM2Q9KOqpwWUR0i4gp5Si\/HDK+3rIcc9VE0pmSXpC0RNLphesi4lZgE0lfyie6fOSSLCX9GPgj8BuSg2wA8GdgWBmKXwd4JSKWlKGsmtdSDaYVDgfujIgFZQinrcr9mFOinJ\/nycDJwB0trB8DjCjj\/qpfRKzQG7Aq8DFwQJFtupAk05np7Y9Al3TdTsB04ETgHWAW8L103RnAImBxuo8jgdOB0QVlDwQCqEsfHw5MAeYCrwPDC5Y\/WvC87YCngQ\/T\/7crWPcgcCbwWFrOPUCvFl5bY\/wnF8S\/H7AX8ArwHvCzgu23Bh4HPki3vQjonK57OH0t89LXe2BB+T8F3gL+3rgsfc766T42Tx\/3BeYAO7UQ7\/3Ad9L7qwALgE\/T\/X2cPr8DcArwGvAucAOwevqclYDR6fIP0veuD3A20AAsTMu5KN0+gA3S+1cCF5N8YOcCTwLrF8S2G\/By+jf5M\/AQcFQLr0PA+el7\/iHwPLBJwfE2CngTeBu4BOja0uttUm5zx1wH4BfA1HR\/VwOrtnD8rZvGPRcYm\/59C4\/XbYF\/p+\/dc4V\/J5Lj7myS425B4\/tW5s\/raOD0ZpZvD7y+ovNHnrcVv0PYA1jSeLC0sM2vgSeANYHe6cFyZrpup\/T5vwY6kSSZ+cBq6frTmxxsTR8vPVjTD8NHwIbpurWBjdP7h5MmS2B14H3g0PR5B6eP1yg4aF8DBqcfsgeBc1t4bY3x\/yqN\/2hgNnAt0B3YmCSBrJduv0X6galLY58EjCwob2lyaVL+b0mSQFcKkmW6zdFpOSsDdwOjivwtZgNbNSl\/epNtRqZ\/r37pPv8KjEnXHQPclu6rY\/p6ehS8b0c1KatpsnyP5AujDrgGuC5d1yv9230zXfdDkoTVUrLcHRgP9CRJnF8A1k7X\/RG4Nf07d0\/jPael19tM2afz+WPsCJKa2XpAN+Bm4O9Nj7\/08ePAeen7tiNJ0hydrqsn+ZLZiyQBfz193Lvg\/XszPWbqgE7NxHY7SaJt7nZ7hs9rS8ly9fR19FjROSSvWx7N8DWAOVG8yTIc+HVEvBMRs0m+vQ8tWL84Xb84Iu4k+UZfpj45klrDJpK6RsSsiJjYzDbfAF6NiL9HxJKIGAO8BOxTsM3\/RsQrkTRXbwCGFtnnYuDsiFgMXEfywf9TRMxN9z8R+BJARIyPiCfS\/b5Bkoi+muE1nRYRn0QzzeeIuAx4laSmtjbw8yJl9ST5ABdzDPDziJgeEZ+QJI9vp10Ai0n+5htEREP6ej4qUV6hmyPiqfR4uYbP3te9gIkRcXO67gKSmnRLFpMkwo0ARcSkiJglSSRfHj+KiPciYi5J99BBrYixqeHAeRExJSI+Bk4FDmraJSJpALAV8Mv0b\/UwSaJu9B2SLpA7I+LTiBgLjEtfe6MrI2JienwsbhpIROwdET1buO29HK+x8ZjouRxl1JQ8kuW7QK8SfWl9SZowjaamy5aW0STZzif5Bm+ViJhH0nQ9Fpgl6Q5JG2WIpzGm+oLHhR\/UUvG8GxEN6f3GZPZ2wfoFjc+XNFjS7ZLekvQRyQe5V5GyAWZHxMIS21wGbAJcmCa4lrxPkmSKWQf4p6QPJH1AUmttIGlu\/52k9nqdpJmSfiepU4nyCrX0vvYFpjWuiKS6s\/REjKSJ6cnDjyV9JSLuJ2niXgy8LelSST1IWi4rA+ML4v9XunxZNXf81pG8H023ez89Dgu3bbQOcEBjXGlsO5B8wTWaRj4aj4kPctr\/CpdHsnycpJm5X5FtZpIcKI0GpMuWxTySD0OjtQpXRsTdEfF1kgPwJZIkUiqexphmLGNMrfEXkrgGRUQP4Gckzchiik4lJakbSdPzb8DpklYvsvnzJN0LxcqeBuzZpNayUkTMSGv\/Z0TEEJJ+371Jzq6XjLOEWSTN\/sbXpMLHEbFxJGfWu0XEI+myCyJiC5Jm62DgJJL+2gUk3S+Nsa8aEY1JeVlibO74XcLnvxAbX8NqklZpsm2jaSTN98L3dZWIOLdgm1J\/67sKvjSa3u5q\/Utb6gvAG61sJdS0FZ4sI+JDkv66iyXtJ2llSZ0k7Snpd+lmY4BfSOotqVe6\/bKOPXsW2FHSAEmrkjSJAJDUJx0ztgrwCUlzvqGZMu4EBqfDneokHQgMIekPqrTuJH1zH6e13uOarH+bpG+sNf4EjI+Io0hOnlxSZNs7+Xyz\/21gjfS9bHQJcLakdSAZbiRpWHp\/Z0lflNQxfR2L+ew9XpbYG90BfDE9huqA42nyRVhI0laStklrtfNIvrAbIuJTki\/I8yWtmW5bL2n3Iq+3lDHAjyStm34x\/Qa4vmnXU0RMJWlWnyGpczp0rrBrZzSwj6TdJXWUtFI6lKkfGUXEngVfGk1ve7b0vPQzuRJJjqhL992xYJOvAsuTbGtOLkOHIuI84MckZwxnk3yDngD8X7rJWSQH0fPAC8CEdNmy7GsscH1a1ng+n+A6kJxVn0lyIuGrwP9rpox3SWpEJ5J0I5wM7B0Rc5Ylplb6CXAISR\/RZSSvpdDpwFVpM+1\/ShWWJrE9SLoeIPk7bC5peAtPuRrYS1JXgIh4iSQZTEn32Zck+d4K3CNpLsnJnm3S568F\/IMkUU4iOfPb+MX3J5K+zfclXVAq9kLpe38A8DuSv8kQkmOmpS6FHiTv3\/skTd13Sc6AQzJyYDLwRNrVcS9pH3gLr7eUK0i6Hx4mGWGxEPh+C9seQvJevQecRvJ+N77GaSTD6X7GZ5+Tk1gxn9vLSGrcB5P0aS\/g8+cNDiYde9teKOnqMWuZpN8A70TEH\/OOpSXpGMPpJEO\/Hsg7nrZM0j7AoRFR8su5LXGytJqVNpWfJKn1nETSFF+vuREAZsurzV0bbu3Kl0nGt84h6evbz4nSKsU1SzOzDFyzNDPLYHknWSirNXr1igHrDMw7jJry6aduGbRWhw6lhqlaU29OfYN358wp6xvXscc6EUuy95rEgtl3R8Qe5YyhNaoqWQ5YZyAPPfZU3mHUlLkLPblSa3VfqaoO+5rw1e23LnuZsWQBXTbMfkJ94bMXl7pyraJ81JhZTgRlnVWuspwszSwfAlQ7XSJOlmaWH9cszcxKEXToWHqzKuFkaWb5cTPczKwE4Wa4mVlpcs3SzCwT1yzNzDJwzdLMrBQPSjczK82D0s3MMnLN0sysFEFHD0o3MyvO4yzNzDJyn6WZWSk+G25mlo1rlmZmGbhmaWZWgnxtuJlZNq5Zmpll4JqlmVkpPhtuZlaa8M9KmJmV5pqlmVk27rM0M8ughmqWtROpmbU9jWMts9yKFqP+kh6QNEnSREk\/TJefLmmGpGfT214FzzlV0mRJL0vavVSorlmaWT5U1j7LJcCJETFBUndgvKSx6brzI2LU53etIcBBwMZAX+BeSYMjoqGlHbhmaWb5KVPNMiJmRcSE9P5cYBJQX+Qpw4DrIuKTiHgdmAxsXWwfTpZmlhtJmW+tKHMgsBnwZLroBEnPS7pC0mrpsnpgWsHTplM8uTpZlnLxBX9km82\/yLZbfIkjvnsICxcuzDukqjNz+jQO2Gc3dtrmS3zty0O5\/JILl6674tKL2XGrTfjal4dy1q9OzTHK6tYej7PkJ3halSx7SRpXcBvxX2VK3YCbgJER8RHwF2B9YCgwC\/hDwe6bimLxus+yiJkzZnDJny\/kqWdepGvXrhw2\/EBuuvE6hh96eN6hVZWOdXX86qzf8sVNN+PjuXPZc+dt2XGnXZk9+23uufM2xj46ni5dujBn9jt5h1qV2u1xJqEOrRo6NCcitmy5OHUiSZTXRMTNABHxdsH6y4Db04fTgf4FT+8HzCy2c9csS2hYsoQFCxawZMkSFiyYz1pr9807pKrTZ621+eKmmwHQrXt3Bg3eiLdmzeDvV1zK8SNPokuXLgD06r1mnmFWtfZ6nJWrGa5kg78BkyLivILlaxdstj\/wYnr\/VuAgSV0krQsMAp4qtg8nyyL61tfz\/ZEnssnggQxet54ePVZll113yzusqjbtzTd48fnn2GyLrZky+VWefPwx9t51B771jV15dsK4vMOrSu35OCtjn+X2wKHA15oME\/qdpBckPQ\/sDPwIICImAjcA\/wH+BRxf7Ew4VDhZStojHcM0WdIpldxXJbz\/\/vvccfutPD\/pNV6eMp358+Zx\/ZjReYdVteZ9\/DEjvnsQp58ziu49etCwZAkffvA+t419hF\/8+hyO+94hRBTtFmqX2vNxVq5kGRGPRoQi4ksRMTS93RkRh0bEF9Pl+0bErILnnB0R60fEhhFxV6lYK5YsJXUELgb2BIYAB6djm2rGg\/ffyzoDB9Krd286derEPvvtz5NPPJ53WFVp8eLFjDjsQPY\/4CD22mc\/ANaqr2fPffZDEpttsRUdOnTgvXfn5Bxp9Wm3x5laectZJWuWWwOTI2JKRCwCriMZ21Qz+vcfwLinnmT+\/PlEBA89cD8bbviFvMOqOhHBT75\/DBsM3ogRx49cunyPvfblsYcfBGDK5FdYtGgxq6\/RK6coq1d7Pc5E9lpla4YOVUolz4Y3N45pm6Ybpaf\/R0By0FSTLbfehmH7f4sdv7wldXV1fGnToRx+5NF5h1V1nn7i39x0\/TVsNGQTdvvKVgD89Je\/5sDvHM6JJ4xgly9vRqfOnfnjXy6vioO+2rTn46yWjgdVqg9J0gHA7hFxVPr4UGDriPh+S8\/ZbIst46HHip6QsibmLlySdwg1p\/tKHjHXWl\/dfmueGT+urJmtbo31osdeZ2Xe\/v3Rw8cXGzpUaZU8alo9jsnM2pdaqllWss\/yaWCQpHUldSa5aP3WCu7PzGpJjZ3gqVjNMiKWSDoBuBvoCFyRjm0yM0OIDh1qZ6h3RTtvIuJO4M5K7sPMalctNcPd021m+amdXOlkaWY5kWuWZmaZOFmamWXgZGlmVkLj5Y61wsnSzPJTO7nSydLMcuITPGZm2ThZmpll0Mrf4MmVk6WZ5cY1SzOzEqplUt+snCzNLDdOlmZmGThZmpllUTu50snSzPLjmqWZWSkelG5mVpqAGsqVTpZmlhfRwYPSzcxKczPczKwUuRluZlaSwM1wM7MsXLM0M8vAfZZmZqXUWJ9lh7wDMLP2KRlnqcy3omVJ\/SU9IGmSpImSfpguX13SWEmvpv+vli6XpAskTZb0vKTNS8XrZGlmOcmeKDM015cAJ0bEF4BtgeMlDQFOAe6LiEHAfeljgD2BQeltBPCXUjtwsjSz3EjZb8VExKyImJDenwtMAuqBYcBV6WZXAful94cBV0fiCaCnpLWL7cN9lmaWD7V66FAvSeMKHl8aEZf+V7HSQGAz4EmgT0TMgiShSloz3awemFbwtOnpslkt7dzJ0sxy0dhn2QpzImLLomVK3YCbgJER8VGR8ptbEcXKdjPczHJTrmZ4UpY6kSTKayLi5nTx243N6\/T\/d9Ll04H+BU\/vB8wsVr6TpZnlpoxnwwX8DZgUEecVrLoVOCy9fxhwS8Hy76ZnxbcFPmxsrrfEzXAzy00Zx1luDxwKvCDp2XTZz4BzgRskHQm8CRyQrrsT2AuYDMwHvldqB1WVLAXU0KWiVWGDnX+cdwg15\/4bz8o7hJqzcFFD+Qst4+S\/EfEoLf9IxS7NbB\/A8a3ZR1UlSzNrPzz5r5lZJv7dcDOzTGooVzpZmllOWj8oPVdOlmaWi2UYlJ4rJ0szy42TpZlZBjWUK50szSw\/rlmamZVSYzOlO1maWS7kcZZmZtnUUK50sjSz\/HSooWzpZGlmuamhXOlkaWb5kKCjr+AxMyutTZzgkdSj2BMj4qPyh2Nm7UkN5cqiNcuJJD\/gU\/hyGh8HMKCCcZlZGyeS4UO1osVkGRH9W1pnZlYONdRlme0HyyQdJOln6f1+kraobFhm1ua14sfKqqFvs2SylHQRsDPJjwFB8uM+l1QyKDNrH8r5U7iVluVs+HYRsbmkZwAi4j1JnSscl5m1cckPFFZBFswoS7JcLKkDyUkdJK0BfFrRqMysXaihXJmpz\/Ji4Cagt6QzgEeB31Y0KjNrF2qpz7JkzTIirpY0Htg1XXRARLxY2bDMrK1rq1fwdAQWkzTFM51BNzMrpXZSZbaz4T8HxgB9gX7AtZJOrXRgZtb2talmOPAdYIuImA8g6WxgPHBOJQMzs7YtORuedxTZZUmWU5tsVwdMqUw4ZtZuVEmNMatiE2mcT9JHOR+YKOnu9PFuJGfEzcyWSw3lyqI1y8Yz3hOBOwqWP1G5cMysPWkTNcuI+NuKDMTM2pc212cpaX3gbGAIsFLj8ogYXMG4qsYHH3zACccdzX8mTkQSf\/7r5Wyz7ZfzDit3\/fr05PIzv0ufNXrwaQRX3PQYF495kC8NrufCnx9Ely6dWNLwKSN\/cz3jJk6lZ\/eu\/PX077Buv158smgxx5x+Df95bVbeLyM3U6e8yq9+eMTSxzOmTeXoH57K7Ldn8ugDd9OpUyfqB6zLz8+9mO49Vs0x0soqZ81S0hXA3sA7EbFJuux04GhgdrrZzyLiznTdqcCRQAPwg4i4u1j5WU7wXAmcBYwC9gS+Rzu63PHkE0ey69d3Z\/SYG1m0aBHz58\/PO6SqsKThU04572aefWk63Vbuwr+v\/Sn3PfkSZ4\/cj7MvvYt7HvsPu+8whLNH7sfuR\/+Jk4\/cnedens6BJ17G4IF9+OMp\/8Nex16Y98vIzTrrDeKq2x4BoKGhgWE7DGHH3b7Bm1Mmc+xPTqOuro6Lf3caV19yHseffEbO0VaGBB3L2wy\/ErgIuLrJ8vMjYtTn960hwEHAxiTDIu+VNDgiGloqPMsA85UbM25EvBYRvyCZhajN++ijj\/j3o49w2PeOBKBz58707Nkz56iqw1tzPuLZl6YD8PH8T3jp9bfo27snEdBjlaQBsmq3rsya\/SEAG623Fg8+9TIAr7zxNuv0XZ01V++eT\/BVZty\/H6J+wEDWrh\/ANl\/5GnV1SR1mk6FbMfutmTlHV1nlnHUoIh4G3su462HAdRHxSUS8DkwGti72hCzJ8hMldeXXJB0raR9gzYwB1bQ3Xp9Cr969OfboI9h+my04\/tijmTdvXt5hVZ0Ba6\/O0A378fSLb3DSqH\/wm5H78epdZ3LOj\/bnVxfeAsALr8xg2C5DAdhy43UYsPbq1PfxFw\/AvXfczNf3\/tZ\/Lb\/9H6PZ9qu7NvOMtqOVg9J7SRpXcBuRcTcnSHpe0hWSVkuX1QPTCraZni5rUZZk+SOgG\/ADYHuS9v8RRZ9B0n8g6R1JNXsd+ZIlS3j2mQkcNeJYHntyPKussgrn\/d5ziBRapWtnxow6ipNG3cTceQsZccBXOPkPNzNoz19y8qib+MtpwwEY9b9j6dl9ZZ647hSOO+irPPfydJY0tJvenBYtXrSIR++\/i6\/tud\/nll\/551F0rKtj933\/J6fIVoxW1iznRMSWBbdLM+ziL8D6wFBgFvCHxl03s20UKyjLRBpPpnfn8tkEwFlcSfP9BzWjvr4f9fX92GrrbQAYtv+3OG+Uk2WjuroOjBl1NNffNY5b7n8OgOF7b8OJv\/sHADeNfYY\/\/+oQAObOW8gxp49e+tyX7jiDN2a8u+KDrjKPP3wvg4dsyuq9Pmus3XnzGB574B4uvPr\/ampoTWsJVXw+y4h4e+n+pMuA29OH04HCn87pBxTt8yg2KP2fFMm0EfHNEkE+LGlgsW2qXZ+11qK+X39eeeVlBg\/ekIceuJ+NvjAk77CqxiWnDefl19\/igtH3L102a\/aHfGWLQTwy\/lV22nowk99MTkKu2q0r8xcuYvGSBr63\/3Y8OmEyc+ctzCv0qjH29n98rgn+xMP3MvrSP3HxNbezUteVc4xsBVgBM6BLWjsiGodd7M9n48dvJZnn4jySEzyDgKeKlVWsZnnR8gaaRdrvMAKgf\/\/q+8HIUef\/iaMOP5RFixYxcN11+culV+QdUlXYbuh6DN97G154ZQZPXHcKAKdddCvHn3ktvz\/p29TVdeCTT5ZwwlljgOQEz+VnHkpDw6e8NOUtjj3jmjzDrwoLF8zn6cce5Kdnnr902R\/OOJnFiz5h5OH7A7Dx0C05uWB9W1PmoUNjgJ1I+janA6cBO0kaSlLxewM4BiAiJkq6AfgPsAQ4vtiZcABFFG2mL2\/wA4HbG8c8lbL5FlvGw\/8umtytid7b\/iDvEGrO\/TeelXcINeeI\/Xdm0gvPlLUeuOYGm8SBv78x8\/YXfXPI+IjYspwxtEbW+SzNzMpKtJHLHc3MKq2WLnfMPOu5pC6tKTjtP3gc2FDSdElHtjY4M2u7Gn9WIustb1muDd8a+BuwKjBA0qbAURHx\/WLPi4iDyxOimbVVVZADM8tSs7yA5OL0dwEi4jnayeWOZlZZ5bzcsdKy9Fl2iIipTTpii55iNzMrJZmirQqyYEZZkuW0tCkekjoC3wdeqWxYZtYe1NJPxWZJlseRNMUHAG8D96bLzMyWSw1VLDNdG\/4OybxvZmZlI1X+2vByynI2\/DKauUY8IrJOj2Rm1qwaypWZmuH3FtxfieRi9GktbGtmllktDR3K0gy\/vvCxpL8DYysWkZm1C4KqGGye1bJc7rgusE65AzGzdkZtrGYp6X0+67PsQPIbF6dUMigzax\/U7ITl1aloskx\/e2dTYEa66NOo5JxuZtZu1NrvhhcdE5omxn9GREN6c6I0s7LpoOy3vGUZQP+UpM0rHomZtTut\/HXHXBX7DZ66iFgC7AAcLek1YB5J7TkiwgnUzJZZrTXDi\/VZPgVsDuxXZBszs2VTJbMJZVUsWQogIl5bQbGYWTvTVi537C3pxy2tjIjzKhCPmbUTbakZ3hHoBjU0EMrMaojo2EZqlrMi4tcrLBIza1eSX3fMO4rsSvZZmplVRJWMn8yqWLLcZYVFYWbtUps4wRMR763IQMysfWlLzXAzs4pqEzVLM7NKq6Fc6WRpZvkQbe\/XHc3Myk9UxQQZWTlZmlluaidV1lYt2MzaEAEdpcy3kuVJV0h6R9KLBctWlzRW0qvp\/6ulyyXpAkmTJT2fZRpKJ0szy42U\/ZbBlcAeTZadAtwXEYOA+\/jsJ3H2BAaltxHAX0oV7mRpZjnJPvFvlr7NiHiY5DfCCg0DrkrvX8VnU04OA66OxBNAT0lrFyvfydLMctF4NjzrDeglaVzBbUSG3fSJiFkA6f9rpsvrgWkF201Pl7XIJ3jMLDetPBs+JyK2LNeum1lW9DfGXLM0s9yoFbdl9HZj8zr9\/510+XSgf8F2\/YCZxQqqqpqlgLqOzt+t8eA\/zso7hJozbe78vEOoOYsaPi1\/oStmnOWtwGHAuen\/txQsP0HSdcA2wIeNzfWWVFWyNLP2o9xX8EgaA+xE0rc5HTiNJEneIOlI4E3ggHTzO4G9gMnAfOB7pcp3sjSz3JSzZhkRB7ew6r+mm4yIAI5vTflOlmaWm7Yy+a+ZWcUkzfDayZZOlmaWmxqaR8PJ0szyIuSapZlZaa5ZmpmV4D5LM7Msss8mVBWcLM0sN06WZmYZ+ASPmVkJwoPSzcwy8e+Gm5ll4Ga4mVkJboabmWXiK3jMzErzOEszs2xqKFc6WZpZPpI+y9pJl06WZpab2kmVTpZmlqcaypZOlmaWGzfDzcwyqJ1U6WRpZnmqoWzpZGlmuRC+3NHMrDQPSjczy6aGcqWTpZnlqIaypZOlmeXEE2mYmWXiPss25J67\/8VPfvxDGhoaOPyIozjp5FPyDqkqzf3oQ35z6g+Y8uokkPjFOReyUteu\/PaXJ7Jg\/sesVT+AX593Kat075F3qFXjjmsv576bryUi2PWbh\/CN4Udz3k+PZeYbrwEwf+5HrNy9B6OuH5tzpJUhaqoV7mRZTENDAyN\/cDx33DWW+n792GHbrdh77335wpAheYdWdc4\/8xS23XEXzrn4KhYvWsTChQv4wWH78\/1TzmTzbbbnthtHM\/ryCznmRz\/PO9Sq8Obkl7jv5ms55+93UNepE2cfP5zNd9iFH\/\/2kqXbXPWHM1i5W9v+clENVS075B1ANXv6qadYf\/0NWHe99ejcuTMHHHgQt992S95hVZ15cz\/imaf\/zb7\/cygAnTp3pnuPVZk6ZTKbbb0dAFtvvxMP\/Ou2PMOsKjNef5VBX9ycLl270rGujiFbbMtTD\/xr6fqI4PGxt7HDHsNyjLLypOy3vDlZFjFz5gz69eu\/9HF9fT9mzJiRY0TVaca0qay2ei\/O\/OnxfHefHTn71B+wYP481h+8EY\/cexcA9911C++85feuUf\/1N2LShCeY+8F7fLJgARMevZ85b81cun7ShCdZdfXerL3OejlGWXlqxa1kWdIbkl6Q9Kykcemy1SWNlfRq+v9qyxprxZKlpP6SHpA0SdJEST+s1L4qJSL+a1ktNRtWlIaGJbw88Tm+ecgRXH3bw3RdeWWu\/usf+fm5F\/GP0Zdz2LCdmD\/vY+o6dco71KrRb71BDDv8eM487mDOPn44AwcPoWNdx6XrH\/3X\/7X5WmWrMmX2j93OETE0IrZMH58C3BcRg4D70sfLpJI1yyXAiRHxBWBb4HhJNdXZV1\/fj+nTpy19PGPGdPr27ZtjRNVpzbX60nuh\/ICrAAAJiklEQVStvmwyNDk+v7bHvrw88TkGrj+YC666matueZDd9vkW\/Qasm3Ok1WWX\/Q\/md2Pu5tdX3Ey3VXuydvr+NCxZwlP338V2u++bc4SVp1b8W0bDgKvS+1cB+y1rQRVLlhExKyImpPfnApOA+krtrxK23GorJk9+lTdef51FixZx4\/XX8Y292\/4B3Fpr9O5Dn7XrmTrlVQCe\/vfDrLvBhrz37mwAPv30U\/734lHsf\/D38gyz6nz43hwAZs+awZP338X2eySf4+effIS+AzdgjT5t+4tZtLrPspekcQW3EU2KDOAeSeML1vWJiFmQ5CRgzWWNd4WcDZc0ENgMeHJF7K9c6urqOP9PF7HPN3anoaGBww4\/giEbb5x3WFXpxF\/9jtN+PILFixdR338gv\/jtxdz1z+v4x+jLAdhpt73Z+9vDc46yuoz6ydHM\/eB96urqOOqUs+nWoycAj919S9tvgqdaWV+cU9C8bs72ETFT0prAWEkvLU9sTam5frmy7kDqBjwEnB0RNzezfgQwAqD\/gAFbvPLa1IrG09Y8N\/WDvEOoOdPmzs87hJrz00P25LX\/PFfWDvtNNt08bvzXI5m3H9K32\/gSyXIpSacDHwNHAztFxCxJawMPRsSGyxJvRc+GS+oE3ARc01yiBIiISyNiy4jYsnev3pUMx8yqTLn6LCWtIql7431gN+BF4FbgsHSzw4BlHvtXsWa4ktPGfwMmRcR5ldqPmdWuDuWrq\/YB\/pmOVqkDro2If0l6GrhB0pHAm8ABy7qDSvZZbg8cCrwg6dl02c8i4s4K7tPMakmZkmVETAE2bWb5u8Au5dhHxZJlRDxKbV36aWYrkGdKNzPLokouY8zKydLMclNDudLJ0sxyVEPZ0snSzHLimdLNzDJxn6WZWQmeKd3MLKsaypZOlmaWmw411A53sjSz3NROqnSyNLO8eFC6mVlWtZMtnSzNLBeNM6XXCidLM8tNDeVKJ0szy49rlmZmGfhyRzOzLGonVzpZmll+aihXOlmaWT4kX8FjZpZN7eRKJ0szy08N5UonSzPLTw21wp0szSwvnindzKykWrvcsUPeAZiZ1QLXLM0sN7VUs3SyNLPcuM\/SzKyEZFB63lFk52RpZvlxsjQzK83NcDOzDGrpBI+HDplZbtSKW8mypD0kvSxpsqRTyh2rk6WZ5adM2VJSR+BiYE9gCHCwpCHlDNXJ0sxyo1b8K2FrYHJETImIRcB1wLByxlpVfZYTJoyf07WTpuYdRzN6AXPyDqLG+D1bNtX6vq1T7gKfmTD+7pU7q1crnrKSpHEFjy+NiEvT+\/XAtIJ104FtljfGQlWVLCOid94xNEfSuIjYMu84aonfs2XTnt63iNijjMU1V\/WMMpbvZriZtQnTgf4Fj\/sBM8u5AydLM2sLngYGSVpXUmfgIODWcu6gqprhVezS0ptYE37Plo3ft2UQEUsknQDcDXQEroiIieXchyLK2qw3M2uT3Aw3M8vAydLMLAMnSzOzDJwsWyBpQ0lfltQpvZTKMvB71TqSNpC0paQuecdixfkETzMkfRP4DTAjvY0DroyIj3INrIpJGhwRr6T3O0ZEQ94xVTtJe5McZ+8CbwGnNb6HVn1cs2xCUifgQODIiNgFuIVksOvJknrkGlyVSj\/0z0q6FiAiGlzDLE7SdsAo4LCI2Bl4Hyj7TDlWPk6WzesBDErv\/xO4HegMHCLV0gx8lSdpFeAEYCSwSNJocMLM6NyIeCa9fxqwupvj1cvJsomIWAycB3xT0lci4lPgUeBZYIdcg6tCETEPOAK4FvgJyWQHSxNmnrFVuSeBm2FpP28XkskqeqTL1sgvNGuOk2XzHgHuAQ6VtGNENETEtUBfYNN8Q6s+ETEzIj6OiDnAMUDXxoQpaXNJG+UbYfVJj6nGPnABHwDvRcRsScOBsyR1zS9Ca8qXOzYjIhZKuoZk1pJT0w\/7J0AfYFauwVW5iHhX0jHA7yW9RHLp2c45h1XVImIJ8LGkaZLOAXYDDo+IBTmHZgWcLFsQEe9Lugz4D0ltaSHwnYh4O9\/Iql9EzJH0PMms1V+PiOl5x1TN0n7wTsBX0v93iYhX843KmvLQoQzSPqVI+y+tBEmrATcAJ0bE83nHUyskHQ48Xe4JIKw8nCytIiStFBEL846jlkhS+ANZtZwszcwy8NlwM7MMnCzNzDJwsjQzy8DJ0swsAyfLNkJSg6RnJb0o6UZJKy9HWTtJuj29v6+kFid4kNRT0v9bhn2cLuknWZc32eZKSd9uxb4GSnqxtTGaFXKybDsWRMTQiNgEWAQcW7hSiVb\/vSPi1og4t8gmPYFWJ0uzWuNk2TY9AmyQ1qgmSfozMAHoL2k3SY9LmpDWQLsBSNpD0kuSHgW+2ViQpMMlXZTe7yPpn5KeS2\/bAecC66e12t+n250k6WlJz0s6o6Csn0t6WdK9wIalXoSko9NynpN0U5Pa8q6SHpH0SjpFHJI6Svp9wb6PWd430qyRk2UbI6mO5DLDF9JFGwJXR8RmwDzgF8CuEbE5yaTGP5a0EnAZsA\/JJXdrtVD8BcBDEbEpsDkwkWQOxtfSWu1JknYjmd5ua2AosIWkHSVtQfJbzpuRJOOtMrycmyNiq3R\/k4AjC9YNBL4KfAO4JH0NRwIfRsRWaflHS1o3w37MSvK14W1HV0nPpvcfAf5GMkvS1Ih4Il2+LTAEeCydlrMz8DiwEfB64\/XI6YxBI5rZx9eA78LS6dc+TC9tLLRbemucp7EbSfLsDvwzIuan+7g1w2vaRNJZJE39biS\/Cd3ohvTy01clTUlfw27Alwr6M1dN9+3Zx225OVm2HQsiYmjhgjQhzitcBIyNiIObbDeUZIalchBwTkT8tck+Ri7DPq4E9ouI59LrpncqWNe0rEj3\/f2IKEyqSBrYyv2a\/Rc3w9uXJ4DtJW0AIGllSYOBl4B1Ja2fbndwC8+\/DzgufW7H9Gc25pLUGhvdDRxR0BdaL2lN4GFgf0ldJXUnafKX0h2YpeSnPoY3WXeApA5pzOsBL6f7Pi7dHkmDlczkbrbcXLNsR9KJZQ8Hxuizny\/4RUS8ImkEcIekOSQzw2\/STBE\/BC6VdCTQABwXEY9LeiwdmnNX2m\/5BeDxtGb7McnUdhMkXU8y4\/xUkq6CUn5JMqP4VJI+2MKk\/DLwEMkco8emc5BeTtKXOSGd9mw2sF+2d8esOE+kYWaWgZvhZmYZOFmamWXgZGlmloGTpZlZBk6WZmYZOFmamWXgZGlmlsH\/B9BYdM\/+yxbdAAAAAElFTkSuQmCC\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre class=\"\">Current Overall accuracy: 0.6226086956521739\r\nTotal Overall Accuracy: 0.6579710144927536<\/pre>\n<\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAIABJREFUeJzt3XecFPX9x\/HX+wAroiBCQEUsiAUVAUXFGkssMWqURENEExRjNIk9xphYItEkRqPRaGw\/e41daQa7goqKxi52FFGwgaAUP78\/Zg7X82537rhldu\/eTx7zYHfKdz47N\/vZ73xn5juKCMzMrLiavAMwM6sGTpZmZhk4WZqZZeBkaWaWgZOlmVkGTpZmZhm0uGQpaWlJd0r6VNJNi1DOEEljmzO2vEjaStLLi7D86ZKOaM6YiqxrlqQ1Fse6mktj9jlJPSWFpLYNTD9Z0tXlibR5SPqBpOvzjmNxyy1ZSvqJpInpl2OqpFGStmyGovcBugIrRsTgphYSEddExE7NEE9ZpV+8tYrNExEPRUTvJpa\/EjAU+Hf6fltJU5pSVj1l3y\/poMJxEdE+Il5vjvKbQ8bP2yz7XKWQ1EXSdZLeS38AHpE0sHZ6RNwB9JG0YY5hLna5JEtJRwH\/AP5MspP1AP4F7NEMxa8GvBIR85uhrKrXUA2mEQ4ERkbEnGYIp6XKfZ9Torm+z+2BJ4D+QCfgCuBuSe0L5rkOGN5M66sOEbFYB2B5YBYwuMg8S5Ik0\/fS4R\/Akum0bYEpwNHAB8BU4GfptFOAucC8dB3DgJOBqwvK7gkE0DZ9fyDwOjATeAMYUjD+4YLltiDZgT5N\/9+iYNr9wJ+AR9JyxgKdG\/hstfEfVxD\/nsCuwCvAR8AJBfNvCowHPknnPQ9YIp32YPpZPk8\/748Lyv8t8D5wVe24dJk103X0S993B6YD2zYQ773AT9PXywJzgK\/S9c1Kl68BjgdeA2YANwKd0mWWAq5Ox3+SbruuwAhgAfBFWs556fwBrJW+vhw4H7g73a6PAWsWxLYT8HL6N\/kX8ABwUAOfQ8DZ6Tb\/FHgW6FOwv50JvA1MAy4Elm7o89Ypt759rgY4EXgrXd+VwPIN7H+rp3HPBO5J\/76F++tmwKPptnum8O9Est+NINnv5tRutzJ9bz8D+he8HwS8sbjzR57D4l8h7AzMr91ZGpjnVGAC0AVYKd1Z\/pRO2zZd\/lSgHUmSmQ10TKefXGdnq\/t+4c6afhk+A3qn07oB66evDyRNliS\/rh8D+6fL7Ze+X7Fgp30NWDv9kt0PnNHAZ6uN\/49p\/AcDHwLXAssB65MkkDXS+funX5i2aewvAkcUlLcwudQp\/y8kSWBpCpJlOs\/BaTnLAGOAM4v8LT4ENqlT\/pQ68xyR\/r1WSdf5b+C6dNohwJ3putqkn6dDwXY7qE5ZdZPlRyQ\/GG2Ba4Dr02md07\/dD9NpvyFJWA0ly+8BTwIrkCTOdYFu6bR\/AHekf+fl0nhPb+jz1lP2yXxzH\/s5MBlYg6SWdgtwVd39L30\/Hjgr3W5bkyTNq9NpK5P8yOxKkoB3TN+vVLD93k73mbZAu3piu4sk0dY33JXxO9uXZJ9cvmBcp\/RzdFjcOSSvYfGvEIYA75eY5zVg1zo7+psFO+8cCpItya\/3Zg3suHXfL9xZSZLlJ8DewNJ1YjiQr5Pl\/sDjdaaPBw4s2GlPLJj2S2B0A5+tNv426fvl0ngGFszzJLBnA8sfAdxa8L6+ZDkXWKrOuLoJ7g7gfyQ1rCWL\/C3mAeuUKOtFYPuC993S5dqSJI5HgQ3rKft+SifLSwqm7Qq8lL4eCowvmCbgnbrlFUz\/LknNfTOgps5yn\/PNGuvmpLWm+j5vPWXX3cfGAb8seN+7YHsU7n89SH7Yli2Y91q+Tpa\/JU2yBdPHAAcUbL9Ty\/x97ZDuJ7+rM75d+jl6lHP9lTTk0WY5A+hcoi2tO8khTK230nELy4hvtg\/NJvkFb5SI+Jzk0PUXwFRJd0taJ0M8tTGtXPD+\/UbEMyMiFqSva9sCpxVMn1O7vKS1Jd0l6X1Jn5G083YuUjbAhxHxRYl5Lgb6AP+MiC+LzPcxSUIvZjXgVkmfSPqEJHkuIDncvorkC359esLgr5LalSivUEPbtTtJcgQgkm\/wwhMxkp5PTx7OkrRVRNxLcoh7PjBN0kWSOpAcuSwDPFkQ\/+h0fFPVt\/+2Jdkedef7ON0PC+ettRowuDauNLYtSX6Mar1DmUhamqSWPSEiTq8zuXaf+KRc6680eSTL8SRV+j2LzPMeyY5Sq0c6rik+J\/ky1PpO4cSIGBMRO5LsgC+RJJFS8dTG9G4TY2qMC0ji6hURHYATSGpDxUSxiWlD\/T+AS4GTJXUqMvuzJM0Lxcp+B9glIlYoGJaKiHcjYl5EnBIR65G0+36fpFZYMs4SppIc9td+JhW+j4j1Izmz3j4iHkrHnRsR\/UkOW9cGjiVpr51D0vxSG\/vyEVGblJsSY33773y++YNY+xk6Slq2zry13iGpWRZu12Uj4oyCeUr9rUcV\/GjUHUYVWW5J4DaSffyQemZZl+Ro77Ni629JFnuyjIhPSdrrzpe0p6RlJLWTtIukv6azXQecKGklSZ3T+Zt67dkkYGtJPSQtD\/yudoKkruk1Y8sCX5I00C+op4yRwNrp5U5tJf0YWI+kPajcliNpm5uV1noPrTN9GknbWGOcAzwZEQeRnDy5sMi8I4Ft6qxvxXRb1roQGCFpNUguN5K0R\/p6O0kbSGqTfo55fL2NmxJ7rbuBDdJ9qC1wGHV+CAtJ2kTSwLRW+znJD\/aCiPiK5AfybEld0nlXlvS9Ip+3lOuAIyWtnv4w\/Rm4oc7REBHxFjAROEXSEumlc7sXzHI1sLuk70lqI2mp9FKmVcgoInYp+NGoO+xS3zLpNvoPyY\/I0HQb1bUN0GCybYlyuXQoIs4CjiI5Y\/ghyS\/o4SS\/ZACnkexEz5K0lzyVjmvKuu4BbkjLepJvJrgakrPq75GcSNiGpL2xbhkzSGpER5M0IxwHfD8ipjclpkY6BvgJScP\/xSSfpdDJwBXpYdqPShWWJrGdSZoeIPk79JM0pIFFrgR2TQ\/JiIiXSJLB6+k6u5Mk3zuAsZJmkpzsqb0u7zskX7zPSA7PH+DrH75zgH0kfSzp3FKxF0q3\/WDgryR\/k\/VI9pmGmhQ6kGy\/j0kOdWeQnAGHpG1wMjAhber4L0k7Y0Oft5TLSJofHiS5wuIL4FcNzPsTkm31EXASyfau\/YzvkFxOdwJff0+Opfzf29ojgJ2ATwqbMwrm2Y\/02tvWQmljrVmDJP0Z+CAi\/pF3LA1JrzGcQnLp1315x9OSSdod2D8iSv44tyROlla10kPlx0gOF48lORRfI3wBvZVBi7s33FqVzUkuM5tO0ta3pxOllYtrlmZmGbhmaWaWwaJ2stCsOnfuHD1W65l3GFVl3vz6ruqwYtq1dR2hsd5+602mT59e6vreRmnTYbWI+dlbTWLOh2MiYufmjKExKipZ9litJw8++njeYVSVqZ+UulHH6uq2wlJ5h1B1tt5i02YvM+bPYcne2U+ofzHp\/FJ3rpVVRSVLM2tNBM3Wq1z5OVmaWT4EqFmP7MvKydLM8uOapZlZKYKaNnkHkZmTpZnlx4fhZmYlCB+Gm5mVJtcszcwycc3SzCwD1yzNzEqprovSqydSM2tZai9KzzoUK0paVdJ9kl5MH1b3m3T8yZLelTQpHXYtWOZ3kiZLerngMSINcs3SzPLTfDXL+cDREfGUpOVIntZ5Tzrt7Ig4s3BmSesB+5I8vK478F9Jaxc8dfVbnCzNLCeCNs1zUXpETCV5WiYRMVPSi3zzUdV17QFcnz4G+g1Jk4FNSZ4+Wy8fhptZPmqvs8w6QGdJEwuG4fUWK\/UENiZ55AjA4ZKelXSZpI7puJX55jPXp1A8uTpZmlmOGtdmOT0iBhQMF327OLUHbgaOSJ9pfgGwJtCXpOb599pZ64mm6GMjfBhuZjlp3rPh6fPObwauiYhbACJiWsH0i\/n6UdhTgFULFl+F5JHYDXLN0szy03xnwwVcCrwYEWcVjO9WMNtewHPp6zuAfSUtKWl1oBdQtOdx1yzNLD\/NV7McBOwP\/E\/SpHTcCcB+kvqSHGK\/CRwCEBHPS7oReIHkTPphxc6Eg5OlmeUlQ40xq4h4mPrbIUcWWWYEMCLrOpwszSw\/VXQHj5OlmeXH94abmZVSXfeGO1maWT6EHythZlaaa5ZmZtm4zdLMLAPXLM3MMnDN0sysBLnN0swsG9cszcxKUxUly+qpA+fkX+edy6b9NmSTjTfg\/H+ek3c4FW3BggXsscPmDP\/p3gCccOSh7P7dgey+3ab8atgQPv98Vs4RVq7WuJ8lj+BR5iFvTpZFvPD8c1x+2SXc\/\/AExj\/xNKNH3s3kya\/mHVbFuuLi81mzV++F70849S\/cee9j3Hnf43RbZRWuvuzCHKOrXK12P5NQTfYhb06WRbz80otssulAlllmGdq2bcuWW23NnbfflndYFen9997l\/v+OZvCQAxeOa79cBwAigi\/nfIHq7RTGWvN+5pplC7Hu+n145OGHmDFjBrNnz2bMmFG8O+Wd0gu2QiP+cBzH\/WEENXXObh7\/m0PYYoPVeX3yK+w\/7NCcoqtsrXk\/c7JMSdo5fSbvZEnHl3Nd5bDOOuty5NHHssdu32Ov3Xdlgw02pG1bnxOr676xo1ix80r02Wjjb00745x\/8\/Azr7Fmr96MvP0\/OURX+VrzfuZkCUhqA5wP7AKsR9Jj8XrlWl+5HPCzYTw8YSJjxt1Px46dWHOtXnmHVHGefGI848bezXYD1uXIXxzAhEce4JjDfr5weps2bdh1j70Zc\/ftOUZZ2VrlfqZGDjkrZ81yU2ByRLweEXOB60me1VtVPvzgAwDeeftt7rj9Vvb50b45R1R5jvn9qTz09KvcN\/FFzr7wCjYbtA1\/O+9S3nrjNSBps7x37EjWWGvtnCOtXK1xPxPZa5WVULMsZ12\/vufyDqw7U\/rs3+EAq67ao4zhNM2QfQfz0UczaNeuHWf945907Nix9EJGRPDbXw9n1szPiAjWWX8DTvlL67gkpila635WCUkwq3Imy0zP5U2f\/XsRQL\/+A4o+tzcPY+99IO8QqsrAQVszcNDWAFx\/57ico6kerXU\/c7JMNPq5vGbWulRTsixnm+UTQC9Jq0taAtiX5Fm9ZmZVd4KnbDXLiJgv6XBgDNAGuCwini\/X+sysughRU1M9l3qX9WKuiBhJkef2mlnrVk2H4a3jylczq0zVkyudLM0sJ3LN0swsEydLM7MMnCzNzEqovd2xWjhZmll+qidXOlmaWU58gsfMLBsnSzOzDCrh2TpZOVmaWW5cszQzK6FSOvXNqnruYjezFqe5ekqXtKqk+yS9KOl5Sb9Jx3eSdI+kV9P\/O6bjJenc9Plgz0rqVypWJ0szy00zPlZiPnB0RKwLbAYclj7z63hgXET0Asal7yF5NlivdBgOXFBqBU6WZpafZurPMiKmRsRT6euZwIskj7bZA7gine0KYM\/09R7AlZGYAKwgqVuxdbjN0sxy08g2y86SJha8vyh9LE3dMnsCGwOPAV0jYiokCVVSl3S2+p4RtjIwtaGVO1maWT4af1H69IgYULRIqT1wM3BERHxWpPxMzwgr5MNwM8uFACn7ULI8qR1JorwmIm5JR0+rPbxO\/\/8gHd\/oZ4Q5WZpZTkRNTfahaElJFfJS4MWIOKtg0h3AAenrA4DbC8YPTc+KbwZ8Wnu43hAfhptZbprxOstBwP7A\/yRNSsedAJwB3ChpGPA2MDidNhLYFZgMzAZ+VmoFTpZmlo+Mh9dZRMTDNHzOfPt65g\/gsMasw8nSzHIhKHl4XUmcLM0sN1V0t6OTpZnlp5ruDXeyNLN8NGOb5eLgZGlmuUius6yebOlkaWY5qa4u2pwszSw3VZQrnSzNLCfypUNmZiW5zdLMLKMqypVOlmaWH9cszcwyqKJcWXnJsqaatl4F2HDn4\/IOoeo8PfIveYdQdebO\/6r5C21857+5qrhkaWatQ23nv9XCydLMcuKL0s3MMqmiXOlkaWY58UXpZmal+aJ0M7OMnCzNzDKoolzpZGlm+XHN0sysFPeUbmZWmnydpZlZNlWUK50szSw\/1dQXhJOlmeWminKlk6WZ5UOCNr6Dx8ystBZxgkdSh2ILRsRnzR+OmbUmVZQri9YsnweC5BbOWrXvA+hRxrjMrIUTyeVD1aLBZBkRqy7OQMys9amiJktqsswkaV9JJ6SvV5HUv7xhmVmLp+Si9KxD3komS0nnAdsB+6ejZgMXljMoM2sdpOxD3rKcDd8iIvpJehogIj6StESZ4zKzFk60vIvS50mqITmpg6QVgTI86s3MWpsqypWZ2izPB24GVpJ0CvAw4GeJmtkia842S0mXSfpA0nMF406W9K6kSemwa8G030maLOllSd8rVX7JmmVEXCnpSWCHdNTgiHiu2DJmZqWU4Q6ey4HzgCvrjD87Is785rq1HrAvsD7QHfivpLUjYkFDhWc6Gw60AeYBcxuxjJlZUWrEUEpEPAh8lHHVewDXR8SXEfEGMBnYtNgCWc6G\/x64jiT7rgJcK+l3GQMyM2vQYrp06HBJz6aH6R3TcSsD7xTMMyUd16AstcSfAptExIkR8XuS7Du0KRGbmdVKzoZnH4DOkiYWDMMzrOYCYE2gLzAV+HvB6uuKYgVlORv+Vp352gKvZ1jOzKxhja8xTo+IAY1ZICKmfb06XQzclb6dAhTepbgK8F6xsop1pHE2SaadDTwvaUz6fieSM+JmZouk3JcOSeoWEVPTt3sBtSen7yBpUjyLpImxF\/B4sbKK1SxrC30euLtg\/IRGR2xmVo\/mvI1R0nXAtiSH61OAk4BtJfUlqei9CRwCEBHPS7oReAGYDxxW7Ew4FO9I49Lm+ABmZvWpbbNsLhGxXz2jG8xjETECGJG1\/JJtlpLWTAtcD1iqYEVrZ11JtXrl5ZcZ+tN9F75\/843XOfGPp3D4r4\/IMarKsErXFbjkT0PpumIHvorgspsf4fzr7gfg0H234Rc\/3pr5C75i9EPP8ftzbmffXQZwxAE7LFx+g17d2Xy\/v\/DsK+\/m9Akqw4IFCxi881Z06dadC6\/8DxHBOX85hdF33Uabmhr2HXoQ+x\/0y7zDLJtK6CAjqywneC4HTgPOBHYBfkYrud1x7d69mfDE00CyU6+1+ir8YI+9co6qMsxf8BXHn3ULk16aQvtlluTRa3\/LuMdeokun5fj+thuwyY9OZ+68+azUsT0A14+ayPWjJgKw\/lrduens4a0+UQJcdcm\/WKNXb2bNmgnArTdczdT33mXkg09RU1PDjOkf5Bxh+UjQpoqSZZZLh5aJiDEAEfFaRJxI0gtRq3LfveNYY4016bHaanmHUhHen\/4Zk16aAsCs2V\/y0hvv032lFRg+eCvO\/L97mDtvPgAffjzrW8v+aOf+3Dj6ycUabyV6\/713eWDcaPb5yQELx11\/5SX88sjjqalJvpordu6SV3iLRTX1OpQlWX6ppK78mqRfSNodaNl\/wXr856brGfyjfUvP2Ar16NaJvr1X4Ynn3mSt1bowaOM1efDKYxh7yW\/ov963O9TfZ6d+3Dh6Yg6RVpbTTzqOY048bWFiBHj7rTcYdcfN7LPzVgwfshdvvj45xwjLr0X1ZwkcCbQHfg0MAg4Gfl5qofpuaq9Wc+fOZeRdd7LX3oPzDqXiLLv0Elx35kEce+bNzPz8C9q2qaFjh2XYeuiZnHD2bVz912\/uKpv0WY3ZX8zjhdemNlBi63DfPaPo1Hkl1t9w42+Mn\/fllyy55FL8Z\/RD7DPkQE486tCcIlw8qqlmmaUjjcfSlzP5ugPgLC6n\/pvaq87Y0aPYqG8\/unbtmncoFaVt2xquO\/Ngbhg1kdvvfQaAd6d9wm3jktcTn3+Lr74KOndsz\/T0cHzw9\/q7Vgk8\/cQE7hs7kgfHjWXul18wa+ZMjjt8GF27dWen3fYAYMddfsDvj2y5yVKoqvqzbLBmKelWSbc0NJQquJE3tVe0m268nsE\/9iF4XReeNISX33ifc6++d+G4O+9\/lm03TS6UWKtHF5Zo13ZhopTED3fcmJvGuL3yqBNO4f4nX2Hc4y\/w9wsuZ+CW2\/DX8y5l+513Z8LDDwDwxPiH6LnGWjlHWkaNqFVWQk4tVrM8b3EEkN7fORxg1R6V98DI2bNnc++4ezj3fD9Jo9AWfddgyPcH8r9X3mXC9ccDcNJ5d3DFbeP598lDmHjTCcydt4CD\/njVwmW27LcW7077hDffnZFX2BXv4MOP4tjDh3HFxeexzLLt+dOZ5+cdUllVQltkVoooeu\/4ohUu9QTuiog+Webv139APDz+ibLF0xKtOPBXeYdQdZ4e6b6rG2ufnbfiuWeeatbM1mWtPvHjv92Uef7zfrjek429N7w5ZbnO0sys2Ynqqlk6WZpZblrcc8MBJC3ZmILTm9rHA70lTZE0rLHBmVnLVftYiaxD3rLcG74pyc3oywM9JG0EHBQRRRvLGrip3cxsoQrIgZllqVmeC3wfmAEQEc\/QCm93NLPm11IuHapVExFv1WmILdrvm5lZKUkXbRWQBTPKkizfSQ\/FQ1Ib4FfAK+UNy8xag2p6VGyWZHkoyaF4D2Aa8N90nJnZIqmiimWme8M\/IHkYuZlZs5Gq697wLGfDL6aeR0RGRJbHUJqZNaiKcmWmw\/D\/FrxeiuQJae80MK+ZWWbVdOlQlsPwGwrfS7oKuKdsEZlZqyCoiIvNs2rK7Y6rA362gpktGrWwmqWkj\/m6zbKGpI\/K48sZlJm1DqJ6smXRZJk+e2cjoPYxfF9FOft0M7NWo7mfG15uRa8JTRPjrRGxIB2cKM2s2dQo+5C3LBfQPy6pX9kjMbNWp5qe7tjgYbikthExH9gSOFjSa8DnJLXniAgnUDNrsmo7DC\/WZvk40A\/YczHFYmatSYX0JpRVsWQpgIh4bTHFYmatTEu53XElSUc1NDEizipDPGbWSrSkw\/A2QHuooguhzKyKiDYtpGY5NSJOXWyRmFmrkjzdMe8osivZZmlmVhYVcv1kVsWS5faLLQoza5VaxAmeiPhocQZiZq1LtR2GV9MjMMyshalJe0vPMpQi6TJJH0h6rmBcJ0n3SHo1\/b9jOl6SzpU0WdKzWe5SdLI0s9w086NwLwd2rjPueGBcRPQCxvF1j2m7AL3SYThwQanCnSzNLBciSUBZh1Ii4kGSLiQL7QFckb6+gq\/vSNwDuDISE4AVJHUrVn5TOv81M1t0YnF0kNE1IqYCRMRUSV3S8SvzzcfjTEnHTW2oICdLM8tNI1NlZ0kTC95fFBEXNeOqi3ZB6WRpZrkQNPYOnukRMaCRq5kmqVtaq+wGfJCOnwKsWjDfKsB7xQpym6WZ5aaZT\/DU5w7ggPT1AcDtBeOHpmfFNwM+rT1cb4hrlmaWk+bt1FfSdcC2JIfrU4CTgDOAGyUNA94GBqezjwR2BSYDs4GflSrfydLMclF7Nry5RMR+DUz61t2I6SNyDmtM+U6WZpabSnhcRFZOlmaWm+pJlRWWLAXUVFM3JBXg7+cfk3cIVefNj2bnHULVmTv\/q+YvdPFcZ9lsKipZmlnr0dxtluXmZGlmuXHN0swsg2pqdXOyNLNcJIfh1ZMtnSzNLDdVdBTuZGlmeRFyzdLMrDTXLM3MSnCbpZlZFovWm9Bi52RpZrlxsjQzy8AneMzMShC+KN3MLJMszwOvFE6WZpYbH4abmZXgw3Azs0x8B4+ZWWm+ztLMLJsqypVOlmaWj6TNsnrSpZOlmeWmelKlk6WZ5amKsqWTpZnlxofhZmYZVE+qdLI0szxVUbZ0sjSzXAjf7mhmVpovSjczy6aKcqWTpZnlqIqypZOlmeXEHWmYmWXiNssWZsGCBQwaOIDuK6\/MLbfflXc4FeGqPx\/Hc4\/cy3IdV+TEq8cA8Plnn3DZHw5nxvvvsuJ3VmbYn85nmQ7LM2fWZ1x+6pF8PO09FsxfwA4\/OZjNdxuc8yfI361XXsjom69BEj17rctRp53D2X88gleff4a2bduxdp+N+fVJZ9K2Xbu8Qy0LUVVH4dTkHUA1OO\/cc+i97rp5h1FRNtt1bw476\/JvjBt71QX0HjCIk2+4j94DBjH26gsAeODmq+jWsxcnXDGKI867jlv+OYL58+bmEHXlmD5tKrdfcwnn3jCWC297kK++WsADo25ju9324eI7H+WCWx9g7pdfMPrmq\/MOtawkZR7y5mRZwpQpUxg96m5+9vOD8g6lovTqO5BlO6zwjXHPPnQPA3fZG4CBu+zNMw+OBZIvxBezPyci+HLObJbpsAI1bXxQs2D+fOZ++QUL5s\/nyzlz6LRSVzbdeoeFyaH3BhszfdrUvMMsKyn7ULosvSnpf5ImSZqYjusk6R5Jr6b\/d2xqrE6WJRx79BGMOP2v1NR4U5Uy8+PpLN+5CwDLd+7CzE9mALDN3kN5\/83JnLDHQEYM3ZnBR\/yx1W\/Pzl27sfeBv2ToDhvzk+02YJnllqP\/oO0WTp8\/bx7j7ryJAVt+N8coy0+NGDLaLiL6RsSA9P3xwLiI6AWMS983Sdn2WEmrSrpP0ouSnpf0m3Ktq1xG3n0XXVbqQr\/+\/fMOpaq98PiDrNJrPf58+2P87vK7ufGsk5jz+cy8w8rVzE8\/YcJ9o\/m\/MRO55t5n+XLObO6986aF088\/7bf06b85ffpvlmOUZdaYTNn0o\/A9gCvS11cAeza1oHL+vM8Hjo6IdYHNgMMkrVfG9TW78Y8+wl133UHvtXoydMi+3H\/fvfxs6E\/zDqtiLdexM59O\/wCAT6d\/wHIrrAjAhLv\/Q99tvockuqzSkxW7rcq0t17LM9TcTZrwIF1X7sEKnTrTtl07tth+N16Y9AQA1\/zrb3z68XSGH3dqzlGWnxrxL4MAxkp6UtLwdFzXiJgKkP7fpamxli1ZRsTUiHgqfT0TeBFYuVzrK4c\/jTid196cwsuT3+TKa65n2+2+y\/9d2bIb3BfFBlvuwGOjbgbgsVE3s+FWOwLQsWt3Xn7yUQA+++hDpr39Op2798gtzkqwUreVeenZJ\/lizmwigkkodP1RAAAIUElEQVSPPcSqa6zN6P9czZOP3Mdv\/\/rvFt9UIRrdZtlZ0sSCYXidIgdFRD9gF5LK2dbNGe9iaWWX1BPYGHhscazPyu+yk37Nq09PYNYnH\/P7PTdnt2FHsNP+h3LpHw7n0btupGPX7hx02vkA7HLgr7hqxDGM2H9nIoI9f\/lb2q\/QKedPkK91NuzPljt+n1\/9aAfatGnLmuv0YZfB+7PXJj3p0m0VjhqyKwBb7LAbQw49Judoy6eRR9fTC9oivyUi3kv\/\/0DSrcCmwDRJ3SJiqqRuwAdNjjUimrpsthVI7YEHgBERcUs904cDwwFW7dGj\/yuvvVXWeFqayx5\/M+8Qqs4ayy+bdwhV59c\/2pFXnp\/UrNfv9NmoX9w0+qHM86\/Xvf2TDSVLScsCNRExM319D3AqsD0wIyLOkHQ80CkijmtKvGWtWUpqB9wMXFNfogSIiIuAiwD69x9Q3sxtZhWlGW937Arcml6P2Ra4NiJGS3oCuFHSMOBtoMl3Q5QtWSqJ+lLgxYg4q1zrMbPqVdNMuTIiXgc2qmf8DJLa5SIrZwvyIGB\/4LvpRaKTJO1axvWZWbUp\/6VDzaZsNcuIeJiK+IhmVoncU7qZWRbuKd3MLJsqypVOlmaWoyrKlk6WZpYT95RuZpaJ2yzNzEqokCuCMnOyNLP8VFG2dLI0s9zUVNFxuJOlmeWmelKlk6WZ5cUXpZuZZVU92dLJ0sxyUdtTerVwsjSz3FRRrnSyNLP8uGZpZpaBb3c0M8uienKlk6WZ5aeKcqWTpZnlQ\/IdPGZm2VRPrnSyNLP8VFGudLI0s\/xU0VG4k6WZ5cU9pZuZlVRttzvW5B2AmVk1cM3SzHJTTTVLJ0szy43bLM3MSkguSs87iuycLM0sP06WZmal+TDczCwDn+AxM8uginKlk6WZ5aiKsqWTpZnlppraLBURecewkKQPgbfyjqMenYHpeQdRZbzNmqZSt9tqEbFScxYoaTTJ581qekTs3JwxNEZFJctKJWliRAzIO45q4m3WNN5ulcv3hpuZZeBkaWaWgZNlNhflHUAV8jZrGm+3CuU2SzOzDFyzNDPLwMnSzCwDJ0szswycLBsgqbekzSW1k9Qm73iqhbdV40haS9IASUvmHYsV5xM89ZD0Q+DPwLvpMBG4PCI+yzWwCiZp7Yh4JX3dJiIW5B1TpZP0fZL9bAbwPnBS7Ta0yuOaZR2S2gE\/BoZFxPbA7cCqwHGSOuQaXIVKv\/STJF0LEBELXMMsTtIWwJnAARGxHfAxcHy+UVkxTpb16wD0Sl\/fCtwFLAH8RKqmHvjKT9KywOHAEcBcSVeDE2ZGZ0TE0+nrk4BOPhyvXE6WdUTEPOAs4IeStoqIr4CHgUnAlrkGV4Ei4nPg58C1wDHAUoUJM8\/YKtxjwC2wsJ13SWA1kh9qJK2YX2hWHyfL+j0EjAX2l7R1RCyIiGuB7sBG+YZWeSLivYiYFRHTgUOApWsTpqR+ktbJN8LKk+5TtW3gAj4BPoqIDyUNAU6TtHR+EVpd7s+yHhHxhaRrgAB+l37ZvwS6AlNzDa7CRcQMSYcAf5P0EtAG2C7nsCpaRMwHZkl6R9LpwE7AgRExJ+fQrICTZQMi4mNJFwMvkNSWvgB+GhHT8o2s8kXEdEnPArsAO0bElLxjqmRpO3g7YKv0\/+0j4tV8o7K6fOlQBmmbUqTtl1aCpI7AjcDREfFs3vFUC0kHAk9ExPN5x2Lf5mRpZSFpqYj4Iu84qokkhb+QFcvJ0swsA58NNzPLwMnSzCwDJ0szswycLM3MMnCybCEkLZA0SdJzkm6StMwilLWtpLvS1z+Q1GAHD5JWkPTLJqzjZEnHZB1fZ57LJe3TiHX1lPRcY2M0K+Rk2XLMiYi+EdEHmAv8onCiEo3+e0fEHRFxRpFZVgAanSzNqo2TZcv0ELBWWqN6UdK\/gKeAVSXtJGm8pKfSGmh7AEk7S3pJ0sPAD2sLknSgpPPS110l3SrpmXTYAjgDWDOt1f4tne9YSU9IelbSKQVl\/V7Sy5L+C\/Qu9SEkHZyW84ykm+vUlneQ9JCkV9Iu4pDURtLfCtZ9yKJuSLNaTpYtjKS2JLcZ\/i8d1Ru4MiI2Bj4HTgR2iIh+JJ0aHyVpKeBiYHeSW+6+00Dx5wIPRMRGQD\/geZI+GF9La7XHStqJpHu7TYG+QH9JW0vqD+wLbEySjDfJ8HFuiYhN0vW9CAwrmNYT2AbYDbgw\/QzDgE8jYpO0\/IMlrZ5hPWYl+d7wlmNpSZPS1w8Bl5L0kvRWRExIx28GrAc8knbLuQQwHlgHeKP2fuS0x6Dh9azju8BQWNj92qfprY2FdkqH2n4a25Mkz+WAWyNidrqOOzJ8pj6STiM51G8PjCmYdmN6++mrkl5PP8NOwIYF7ZnLp+t27+O2yJwsW445EdG3cESaED8vHAXcExH71ZmvL0kPS81BwOkR8e866ziiCeu4HNgzIp5J75vetmBa3bIiXfevIqIwqSKpZyPXa\/YtPgxvXSYAgyStBSBpGUlrAy8Bq0taM51vvwaWHwccmi7bJn3MxkySWmOtMcDPC9pCV5bUBXgQ2EvS0pKWIznkL2U5YKqSR30MqTNtsKSaNOY1gJfTdR+azo+ktZX05G62yFyzbEXSjmUPBK7T148vODEiXpE0HLhb0nSSnuH71FPEb4CLJA0DFgCHRsR4SY+kl+aMStst1wXGpzXbWSRd2z0l6QaSHuffImkqKOUPJD2Kv0XSBluYlF8GHiDpY\/QXaR+kl5C0ZT6Vdnv2IbBntq1jVpw70jAzy8CH4WZmGThZmpll4GRpZpaBk6WZWQZOlmZmGThZmpll4GRpZpbB\/wNx1tYW2djDdwAAAABJRU5ErkJggg==\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre class=\"\">Current Overall accuracy: 0.6591304347826087\r\nTotal Overall Accuracy: 0.6582608695652173\r\n<\/pre>\n<\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAIABJREFUeJzt3XmcFOW59vHfNQMiCojIIiiLC6Bo4oJRo8Y1QcQFYyQuuEbFuCSaqIn6atSoiRqjJx41xESPGvccNeKG+x43MKgQXMAVQRQVBQSF8X7\/qBps5sx01+A01T1cXz71obuWp+6uqb77qaeqnlJEYGZmxdXkHYCZWTVwsjQzy8DJ0swsAydLM7MMnCzNzDJwsjQzy6DVJUtJ7SXdKelTSf\/4BuWMlHR\/S8aWF0nfk\/TqN1j+95KOb8mYiqxrrqS1l8W6Wkpz9jlJ\/SSFpDZNTD9T0nXlibRlSNpD0k15x7Gs5ZYsJe0vaVz65Zgh6V5J27RA0XsDPYDVImLE0hYSEddHxJAWiKes0i\/eusXmiYgnImLgUpbfDTgI+Ev6fntJ05amrEbKflTS4YXjIqJDRLzREuW3hIyft0X2uUoi6RFJH0r6TNKLkobXT4uIMcCGkr6dY4jLXC7JUtIvgf8Cfkeyk\/UBLgeGF1suo77AaxGxqAXKqnpN1WCa4RDgnoiY3wLhtFa573NKtOT3+TigZ0R0AkYB10nqWTD9xnT88iMilukArALMBUYUmacdSTKdng7\/BbRLp20PTANOAD4AZgCHptPOAr4EFqbrOAw4E7iuoOx+QABt0veHAG8Ac4A3gZEF458sWG4r4Hng0\/T\/rQqmPQqcDTyVlnM\/0LWJz1Yf\/68K4t8TGAa8BnwMnFow\/+bA08DsdN5LgRXSaY+nn2Ve+nn3KSj\/18D7wN\/rx6XLrJOuY9P0fS9gFrB9E\/E+DByQvl4ZmA98la5vbrp8DXAyMBX4CLgF6JIusyJwXTp+drrtegDnAnXAgrScS9P5A1g3fX01cBlwd7pdnwXWKYhtCPBq+je5HHgMOLyJzyHg4nSbfwq8BGxYsL9dCLwDzARGA+2b+rwNym1sn6sBTgPeTtd3LbBKE\/vfWmncc4AH0r9v4f66JfCvdNu9WPh3ItnvziXZ7+bXb7cyfGc3T\/9OmxeM2xp4c1nnjzyHZb9CGAosqt9Zmpjnt8AzQHegW7qznJ1O2z5d\/rdAW5Ik8zmwajr9zAY7W8P3i3fW9MvwGTAwndYT2CB9fQhpsgS6AJ8AB6bL7Ze+X61gp50KDEi\/ZI8C5zXx2erj\/00a\/xHAh8ANQEdgg3THXDudf3D6hWmTxj4ZOL6gvMXJpUH555MkgfYUJMt0niPSclYC7gMuLPK3+BD4ToPypzWY5\/j077Vmus6\/ADem044E7kzXVZt+nk4F2+3wBmU1TJYfk3xZ2wDXAzel07qmf7u90mnHkSSsppLlzsB4oDNJ4lyfpOYEyY\/xmPTv3DGN9\/dNfd5Gyj6TJfexnwBTgLWBDsBtwN8b7n\/p+6eBi9Ltti1J0rwunbYGyY\/MMJIE\/IP0fbeC7fdOus+0Ado2EttdJIm2seGuEp\/rLpJ9MYCxQE3BtC7p+E7LOofkNSz7FcJI4P0S80wFhjXY0d8q2HnnU5BsSX69t2xix234fvHOSpIsZwM\/Ato3iOEQvk6WBwLPNZj+NHBIwU57WsG0o4GxTXy2+vhr0\/cd03i2KJhnPLBnE8sfD9xe8L6xZPklsGKDcQ0T3BjgZZIaVrsif4uFwHolypoM7FTwvme6XBuSxPEv4NuNlP0opZPl3wqmDQNeSV8fBDxdME3Auw3LK5i+I0nNfUuW\/NKLpGZeWGP9LmmtqbHP20jZDfexh4CjC94PLNgehftfH5IftpUL5r2Br5Plr0mTbMH0+4CDC7bfb8v8fW0L7AL8opHxAfQp5\/oracijzfIjoGuJtrReJIcw9d5Oxy0uI5ZsH\/qc5Be8WSJiHsmh60+BGZLulrRehnjqY1qj4P37zYjno4ioS1\/XtwXOLJg+v355SQMk3SXpfUmfkbTzdi1SNsCHEbGgxDx\/BTYE\/jsivigy3yckCb2YvsDtkmZLmk2SPOtIDrf\/TvIFv0nSdEkXSGpborxCTW3XXiTJEYBIvsGLT8RImpSePJwr6XsR8TDJIe5lwExJV0jqRHLkshIwviD+sen4pdXY\/tuGZHs0nO+TdD8snLdeX2BEfVxpbNuQ\/BjVe5cyioiFEXEvsLOkPQom1e8Ts8u5\/kqSR7J8mqRqv2eReaaT7Cj1+qTjlsY8ki9DvdULJ0bEfRHxA5Id8BWSJFIqnvqY3lvKmJrjzyRx9Y+ksf1UktpQMVFsoqQOJIeeVwJnSupSZPaXSJoXipX9LrBLRHQuGFaMiPfSL9tZETGIpN13N5JaYck4S5hBcthf\/5lU+D4iNojkzHqHiHgiHXdJRAwmOWwdAJxE0l47n6T5pT72VSKiPikvTYyN7b+LWPIHsf4zrCpp5Qbz1nuXpGZZuF1XjojzCuYp9be+t+BHo+FwbzM+UxuS9u5665Mc7X3WjDKq2jJPlhHxKUl73WWS9pS0kqS2knaRdEE6243AaZK6Seqazr+0155NALaV1EfSKsAp9RMk9UivGVsZ+IKkgb6ukTLuAQaklzu1kbQPMIikTafcOpK0zc1Na71HNZg+k6RtrDn+BIyPiMNJTp6MLjLvPcB2Dda3Wrot640GzpXUF5LLjeovNZG0g6RvSapNP8dCvt7GSxN7vbuBb6X7UBvgGBr8EBaS9B1JW6S12nkkP9h1EfEVyQ\/kxZK6p\/OuIWnnIp+3lBuBX0haK\/1h+h1wc4OjISLibWAccJakFdJL53YvmOU6YHdJO0uqlbRieinTmmQUEbsU\/Gg0HHZpbBlJ66Xfx\/bpd\/MAkvbUxwpm2w5oTrKterlcOhQRFwG\/JDlj+CHJL+ixwD\/TWc4h2YleImlXeyEdtzTregC4OS1rPEsmuBqSs+rTSU4kbEfS3tiwjI9IakQnkDQj\/ArYLSJmLU1MzXQisD9Jw\/9fST5LoTOBa9LDtB+XKixNYkNJmh4g+TtsKmlkE4tcCwyT1B4gIl4hSQZvpOvsRZJ8xwD3S5pDcrJni3T51YH\/JUmUk0m+cPU\/fH8C9pb0iaRLSsVeKN32I4ALSP4mg0j2maaaFDqRbL9PSA51PyI5Aw5J2+AU4Jm0qeNBknbGpj5vKVeRND88TnKFxQLgZ03Muz\/JtvoYOINke9d\/xndJLqc7la+\/JydR\/u+tSParD9L1HgfsExEvFMyzH+m1t8sLpY21Zk2S9Dvgg4j4r7xjaUp6jeE0kku\/Hsk7ntZM0u7AgRFR8se5NXGytKqVHio\/S9LmeBLJofja4QvorQxa3b3htlz5LsllZrNI2vr2dKK0cnHN0swsA9cszcwy+KadLLSo1bp2jb59++UdRlWp+8pHBs1Vo1KXqVpD77zzFh\/NmtWiG662U9+IRdlbTWL+h\/dFxNCWjKE5KipZ9u3bj8eeei7vMKrKZ\/MX5h1C1Vl5xYra7avCDltvUXqmZopF82k3MPsJ9QUTLit151pZea8xs5wIWrRXufJysjSzfAiooiYRJ0szy49rlmZmpQhqavMOIjMnSzPLjw\/DzcxKED4MNzMrTa5Zmpll4pqlmVkGrlmamZXii9LNzErzRelmZhm5ZmlmVoqg1helm5kV5+sszcwyqqI2y+pJ62bWyqRnw7MOxUqSekt6RNJkSZMkHZeOP1PSe5ImpMOwgmVOkTRF0qsFz4lvkmuWZpaflqtZLgJOiIgXJHUExkt6IJ12cURcWDizpEHAvsAGQC\/gQUkDIqKuqRW4Zmlm+WmhmmVEzIiIF9LXc4DJwBpFFhkO3BQRX0TEm8AUYPNi63CyNLN8SM0boKukcQXDqMaLVT9gE5JnygMcK+klSVdJWjUdtwbwbsFi0yieXH0YbmY5at7Z8FkRsVnR4qQOwK3A8RHxmaQ\/A2cDkf7\/R+AnJOfiGyr69D8nSzPLTwueDZfUliRRXh8RtwFExMyC6X8F7krfTgN6Fyy+JjC9WPk+DDeznLTo2XABVwKTI+KigvE9C2b7ITAxfT0G2FdSO0lrAf2Boo+Wdc3SzPIhWvKxElsDBwIvS5qQjjsV2E\/SxiSH2G8BRwJExCRJtwD\/ITmTfkyxM+HgZGlmuWm5Xoci4kkab4e8p8gy5wLnZl2Hk6WZ5aeK7uBxsjSz\/PjecDOzDFyzNDMrQe4p3cwsG9cszcxKUxUly+qpAy8jRx95GGv3WZ0tBn978biPP\/6Y4bsOYeMNBzJ81yF88sknOUZYeaZPe5cRewxh+y02YsfvbsLfRl8KwH8mvsQeQ7Zjp60Hc8h+ezHns89yjrSyHHvk4fTv25PvbrbR4nHnnvUbtt58E763xWD22n0oM6YXvamkqiWP4FHmIW9Olg2MPPBgbrtjyUuzLr7wfLbbficmTHyV7bbfiYsvPD+n6CpTbZs2\/Obs83n02RcZc\/\/jXHPlaF57ZTInHXcUp5xxNg89NZ6hu+7B6P++qHRhy5H9DjyI\/\/3n3UuM+9kvTuSp5\/7NE8+OZ+ddduWC35+TU3TLgIRqsg95c7JsYOtttmXVLl2WGHf3XWPY\/4CDANj\/gIO468478gitYvVYvSff2mgTADp07Ej\/Aevx\/oz3mPr6a2y51fcA2Hb7nbjnzn\/mGWbFaWxf69Sp0+LX8+bNq4gaVTlVU83SbZYZfPjBTFbvmdxiunrPnsz68IOcI6pc777zFhNfmsAmgzdn4PobcP+9d7HzsN25647bmD59Wt7hVYWzzziNm264jk6rrMKd9z6YdzhlVQlJMKuy1iwlDU27bJ8i6eRyrsvyN2\/uXEYdvB9n\/u5COnbqxB\/\/+y9c87fR7LLDd5k7dw5t266Qd4hV4fSzzmHS628xYp\/9+Ovoy\/IOp6yqqWZZtmQpqRa4DNgFGERyQ\/ugcq2vnLp178H7M2YA8P6MGXTt1j3niCrPwoULGXXwvvxw730ZtvueAKw7YCA33HY39z7yNHv+aB\/6rrV2zlFWl7332Y8xd9yedxjlo2YOOStnzXJzYEpEvBERXwI3kXTlXnWG7bo7N1x3LQA3XHctu+62R84RVZaI4MSfH8m6A9Zj1DHHLR5f31zx1Vdf8ac\/\/p4DDzk8rxCrxtQpry9+PfbuOxkwYGCO0ZSXyF6rrISaZTnbLBvrtn2LhjOlXcOPAujdu08Zw8nm0IP258knHuOjWbNYb50+nHr6GfzixF9zyAH7cu01V9G7dx+uuf7mvMOsKM8\/+y9uvfkG1hu0IUO2TR5j8uvTf8ubU6dwzZWjAdhltz3ZZ+TBeYZZcQ47eCRPPf4YH300iw3W7cvJp53BA\/fdy+uvv0ZNTQ29e\/fhoksuzzvMsqqEJJiVIor2pL70BUsjgJ0j4vD0\/YHA5hHxs6aW2XTwZvHYU0X737QGPpu\/MO8Qqs7KK\/q8ZnPtsPUW\/PuFcS2a2dqstnZ0Gpb90qhPrhs5vtRjJcqpnHtNs7ttN7PlSzXVLMvZZvk80F\/SWpJWIHlG75gyrs\/MqkmVneApW80yIhZJOha4D6gFroqISeVan5lVFyFqaqrnvpiyNt5ExD0U6dbdzJZv1XQY7pZuM8tP9eRKJ0szy4lcszQzy8TJ0swsAydLM7MS6m93rBZOlmaWn+rJlU6WZpYTn+AxM8vGydLMLINKeLZOVk6WZpYb1yzNzEqolE59s3KyNLPcOFmamWXgZGlmlkX15EonSzPLTzXVLKun500za13Ucs8Nl9Rb0iOSJkuaJOm4dHwXSQ9Iej39f9V0vCRdImmKpJckbVoqXCdLM8uFACn7UMIi4ISIWB\/YEjhG0iDgZOChiOgPPJS+B9gF6J8Oo4A\/l1qBk6WZ5UTU1GQfiomIGRHxQvp6DjCZ5HHcw4Fr0tmuAfZMXw8Hro3EM0BnST2LrcNtlmaWm2a2WXaVNK7g\/RURcUUjZfYDNgGeBXpExAxIEqqk7ulsawDvFiw2LR03o6mVO1maWT6yHV4XmlXqueGSOgC3AsdHxGdFknFjE6JY2U6WZpYLQcnD62aVJ7UlSZTXR8Rt6eiZknqmtcqewAfp+GlA74LF1wSmFyvfbZZmlpuWOsGjpAp5JTA5Ii4qmDQGODh9fTBwR8H4g9Kz4lsCn9YfrjfFNUszy00LXme5NXAg8LKkCem4U4HzgFskHQa8A4xIp90DDAOmAJ8Dh5ZagZOlmeWj+W2WTYqIJ2n6fqCdGpk\/gGOasw4nSzPLRXKdZfXcweNkaWY5cRdtZmaZVFGudLI0s5yoZS8dKjcnSzPLhdsszcwyqqJc6WRpZvlxzdLMLIMqypWVlyzb1FbR1qsA6+54Qt4hVJ0nbjs37xCqzhcL61q+ULlmaWZWUn3nv9XCydLMcuKL0s3MMqmiXOlkaWY58UXpZmal+aJ0M7OMnCzNzDKoolzpZGlm+XHN0syslBbsKX1ZcLI0s1zI11mamWVTRbnSydLM8lNTRdnSydLMclNFudLJ0szyIUGt7+AxMyutVZzgkdSp2IIR8VnLh2Nmy5MqypVFa5aTgCC5hbNe\/fsA+pQxLjNr5URy+VC1aDJZRkTvZRmImS1\/qqjJkposM0naV9Kp6es1JQ0ub1hm1uopuSg965C3kslS0qXADsCB6ajPgdHlDMrMlg9S9iFvWc6GbxURm0r6N0BEfCxphTLHZWatnGh9F6UvlFRDclIHSasBX5U1KjNbLlRRrszUZnkZcCvQTdJZwJPA+WWNysyWC9XUZlmyZhkR10oaD3w\/HTUiIiaWNywza+1a6x08tcBCkkPxTGfQzcxKqZ5Ume1s+P8DbgR6AWsCN0g6pdyBmVnr15KH4ZKukvSBpIkF486U9J6kCekwrGDaKZKmSHpV0s6lys9SszwAGBwRn6crOBcYD\/w+w7JmZo1Kzoa3aJFXA5cC1zYYf3FEXLjEuqVBwL7ABiQVwQclDYiIuqYKz3JI\/TZLJtU2wBsZljMza1oLX5QeEY8DH2dc+3Dgpoj4IiLeBKYAmxdboFhHGheTtFF+DkySdF\/6fgjJGXEzs2+kmSe5u0oaV\/D+ioi4IsNyx0o6CBgHnBARnwBrAM8UzDMtHdekYofh9cf9k4C7C8Y\/08i8ZmbN1sxLgmZFxGbNXMWfgbNJKnpnA38EfkLj55aiWEHFOtK4splBmZllVoY2y\/8jImYuXp\/0V+Cu9O00oLCzoDWB6cXKKnmCR9I6wLnAIGDFgiAGZA+5eq3Xfy06duhITW0tbdq04alnns87pIqwZo\/O\/O3sg+ixWie+iuCqW5\/ishsf5e\/nHUr\/fj0A6NyxPbPnzGfLfc9jsw36cunp+wHJode5o+9hzCMv5fkRcjfns9mcc\/LPmfraZCRx+vmX8tSj9\/P4A\/egmhq6rNaNM\/5wOd169Mw71LIp98XmknpGxIz07Q\/5+oh5DMmVPReRnODpDzxXrKwsZ8OvBs4BLgR2AQ5lObvd8d4HHqZr1655h1FRFtV9xckX3caEV6bRYaV2\/OuGX\/PQs69w4Mn\/s3ie8375Qz6dOx+ASVOns\/XIC6ir+4rVu3bi2ZtP4e7HJ1JXt1ztSkv4429P5rvbfZ\/zL7+WhV9+yYIFn7N2\/\/U46penAXDT1aP52yUXcMq5F+ccaXlIUNuCyVLSjcD2JG2b04AzgO0lbUxyiP0WcCRAREySdAvwH2ARcEyxM+GQLVmuFBH3SbowIqYCp0l6Ymk\/kLUO78\/6jPdnJZ3lz\/38C1558316devMK2+8v3ieH\/1gU4YeeQkA8xcsXDy+3QptiSjaPNTqzZ3zGf9+7l+c8Yc\/A9B2hRVou8KS\/dPM\/\/zzirjNr5xa8uNFxH6NjG6yOTEiziU5as4kS7L8QslfbKqknwLvAd2zrqDaSWL3YTsjicOOGMVhh4\/KO6SK06dnFzYeuCbPT3xr8bitN12HmR\/PYeo7Hy4e950N+zL6zAPo07MLh512zXJdq3zv3bfo3KUrZ\/3qaF6fPJH1N9yYE35zHu1XWpnLLzybu2+\/iQ4dOzH6+jvzDrWsqunHIMt1lr8AOgA\/B7YGjiA5m1RUY1fTV6OHHn2Sp58bzz\/vvIcr\/nw5Tz7xeN4hVZSV26\/AjRcezkkX3sqceQsWj\/\/x0M34x9hxS8z7\/MS3Gbz3uWxzwAWc9JMhtFth+X1eXt2iOl6d9CJ7jzyM6+96ghVXWomrRyeH20efeDp3PzWJoXuM4JZrs1wZU72qqT\/LkskyIp6NiDkR8U5EHBgRe0TEUxnKvhoY+o0jzFmvXr0A6N69O7sP35NxzxdtA16utGlTw40XHsHN947jjodfXDy+traG4TtuxP\/e90Kjy7365kzmzf+SDdbttaxCrTjde\/ai++q92HDj5EqYnYYO59WJS57wGjp8bx6+r\/XWLIWoUfYhb00mS0m3S7qtqaFUwc28mr4izZs3jzlz5ix+\/dCDDzBogw1zjqpyjD5jJK+++T6XXPfwEuN33GIgr701k\/c+mL14XN9eq1Fbm+xufXquyoB+PXh7+kfLNN5K0rVbD3r0XJO33ngdgOf\/9Rhr9R\/IO29OXTzP4w\/eS7+1++cVYvk1o1ZZAbmyaJvlpcsiAEmjgFEAvftU1gMjP5g5k31H7AXAokWL+PG++zFk56qvLLeIrTZem5G7bcHLr73HMzedDMAZl47hvif\/w4idB3PL2PFLzr\/J2px46BAWLqrjq6+C4353Mx\/NnpdH6BXjxDPP5zfHH8HChV+yRp9+\/OaCyznn5J\/x9ptTqJFYfY3enHJO6zwTXq+a2ixVzrOSkvoBd0VEpurYpoM3C1\/H2DxdNv9Z3iFUnSduy3wC1FIH7bE9\/3n53y2a2bqvu2Hs84d\/ZJ7\/0r0GjV+KO3hazPLbwm5muRLVVbN0sjSz3FRRR+nZez2X1K45BadX0z8NDJQ0TdJhzQ3OzFqv+sdKZB3yluXe8M1JroJfBegjaSPg8Igo2ljWxNX0ZmaLVUAOzCxLzfISYDfgI4CIeBHYoZxBmdnyobVcOlSvJiLebtAQW\/SGczOzUpIu2iogC2aUJVm+mx6Kh6Ra4GfAa+UNy8yWB9X0qNgsyfIokkPxPsBM4MF0nJnZN1JFFcvSyTIiPiB5CpqZWYtRhdzznVWWs+F\/pZFnU0SE+yozs2+kinJlpsPwBwter0jSNfu75QnHzJYn1XTpUJbD8JsL30v6O\/BA2SIys+WCoCIuNs9qaW53XAvo29KBmNlyRq2sZinpE75us6wh6aPy5HIGZWbLBzX6+O7KVDRZps\/e2YjkuTsAX8Xy\/qQpM2sRy+K54S2p6DWhaWK8PSLq0sGJ0sxaTI2yD3nLcgH9c5I2LXskZrbckZR5yFuTh+GS2kTEImAb4AhJU4F5JLXniAgnUDNbatV2GF6szfI5YFNgz2UUi5ktTyqkN6GsiiVLAUTE1CLzmJkttdZyu2M3Sb9samJEXFSGeMxsOdGaDsNrgQ5QRRdCmVkVEbWtpGY5IyJ+u8wiMbPlSvJ0x7yjyK5km6WZWVlUyPWTWRVLljstsyjMbLnUKk7wRMTHyzIQM1u+tKbDcDOzsmoVNUszs3KrolxZVQ9XM7NWRCQJKOtQsjzpKkkfSJpYMK6LpAckvZ7+v2o6XpIukTRF0ktZ+r9wsjSzfKjFO9K4GhjaYNzJwEMR0R94iK\/74t0F6J8Oo4A\/lyrcydLMcqNmDKVExOMknZMXGg5ck76+hq\/7uhgOXBuJZ4DOknoWK99tlmaWC0Fz7+DpKmlcwfsrIuKKEsv0iIgZABExQ1L3dPwaLPngxWnpuBlNFeRkaWa5aeYJnlkRsVlLrbqRcUU7N3eyNLOcLJNOfWdK6pnWKnsCH6TjpwG9C+ZbE5herCC3WZpZLlr6bHgTxgAHp68PBu4oGH9QelZ8S+DT+sP1prhmaWa5acmapaQbge1J2janAWcA5wG3SDoMeAcYkc5+DzAMmAJ8DhxaqnwnSzPLTUsehEfEfk1M+j\/9XKQPXzymOeVXVLJM7hWtokv6K8Dt152RdwhVZ8rsuXmHUHUW1H3V8oWqur7vFZUszWz5Ud9mWS2cLM0sN65Zmpll0Fo6\/zUzK5vkMLx6sqWTpZnlpoqOwp0szSwvQq5ZmpmV5pqlmVkJbrM0M8tCrlmamWXiZGlmloFP8JiZlSB8UbqZWSZ+briZWQY+DDczK8GH4WZmmfgOHjOz0nydpZlZNlWUK50szSwfSZtl9aRLJ0szy031pEonSzPLUxVlSydLM8uND8PNzDKonlTpZGlmeaqibOlkaWa5EL7d0cysNF+UbmaWTRXlSidLM8tRFWVLJ0szy4k70jAzy8Rtlq3EggUL+P4O2\/LlF1+wqG4RP9xrb04\/46y8w6pIt187mrG3Xo8k+vVfn1+e8ycuO+dkXp80gYhgjX7rcMK5l9B+pQ55h1ox7r3hbzx8+41EBDv+cH+GjTwcgLE3XcX9N19NTW0bNtlmR0Yef1rOkZaHqKqjcCfLYtq1a8fYBx6mQ4cOLFy4kB2324YhO+\/CFltumXdoFWXWzBnccf3f+MsdT9Buxfb87oTDeezefzLq12ezcoeOAFxxwencecNV\/Pjwn+ccbWV4d8orPHz7jZxz7V20aduW8449gE2+tyMfz5zB+Efv5\/ybH6DtCu349ONZeYdaVqqiqqWTZRGS6NAhqQktXLiQRQsXVtUfd1mqW7SIL79YQJs2bfli\/ny6dOuxOFFGBF8sWFBd1Ygye+\/NKfT\/1ia0a98egPUHb8nzD4\/ljckvscehx9B2hXYArNKla55hll01fZ1q8g6g0tXV1bHF4I3p06s7O37\/B2y+xRZ5h1RxuvboyY8OOZqDvr8J++\/wLVbq2JHBW+8AwEWn\/Zz9t9uAaW9OYY\/9D8850srRe52BTH7hWebM\/oQv5s9nwpMP89HM6bz\/9hu88sKznHbQbpx1+I+YOmlC3qGWlZoxlCxLekvSy5ImSBqXjusi6QFJr6f\/r7q0sZYtWUrqLekRSZMlTZJ0XLnWVU61tbU8O34CU96axrjnn2PSxIl5h1Rx5nw6m2ceGcv\/3DeO6x9+iS\/mf87Dd\/4DgF+yseTZAAAJnElEQVSecwnXPfIyvdfuz+Nj78g50sqxxtr92eOQo\/nd0ftx3rEH0GfAIGpr21BXV8e8OZ9y9jV3MvL40\/jTr48iIvIOtzyakymz10B3iIiNI2Kz9P3JwEMR0R94KH2\/VMpZs1wEnBAR6wNbAsdIGlTG9ZVV586d2Xa77bn\/\/rF5h1JxJjzzOD3W6EPnLl1p07YtW+20K\/+Z8Pzi6bW1tWw7dE+eeuCuHKOsPDvsuR+\/v2EsZ1x5Kx06dWb1PmvRpfvqbL7jLkhi3Q03QTU1zJn9cd6hlo2a8W8pDQeuSV9fA+y5tAWVLVlGxIyIeCF9PQeYDKxRrvWVw4cffsjs2bMBmD9\/Pg8\/9CADB66Xc1SVp1vPNXjlpfEsmP85EcGEZ5+g99oDmP7OG0DSZvnso\/ex5lrr5hxpZak\/eTNrxns8\/8i9bDV0OJvtMJRJzz8FwIy332DRwi\/p2LlLnmGWjUjaLLMOQFdJ4wqGUQ2KDOB+SeMLpvWIiBmQ5CSg+9LGu0xO8EjqB2wCPLss1tdS3p8xgyN+cjB1dXV8FV\/xo71\/zLBdd8s7rIqz3rcHs80PduNnP\/4+tbVtWGe9DdllxIGc8pO9+HzeXCKCtQYO4tjT\/5B3qBXl4hNHMffTT6ht04ZDf30uHTp1Zofh+zD6zBM4acROtGnblqPO+q9WfVKxmZ9sVsHhdWO2jojpkroDD0h65ZvE1pDK3R4iqQPwGHBuRNzWyPRRwCiA3n36DH5t6ttljae1efTVD\/MOoerM\/vLLvEOoOqeOHMYb\/3mxRbP2hhttGv8Y+0Tm+Qf16jC+RLJcTNKZwFzgCGD7iJghqSfwaEQMXJp4y3o2XFJb4Fbg+sYSJUBEXBERm0XEZt26ditnOGZWYVqqzVLSypI61r8GhgATgTHAwelsBwNLfZaxbIfhSo4drgQmR8RF5VqPmVWvmparq\/YAbk+bLNoAN0TEWEnPA7dIOgx4BxixtCsoZ5vl1sCBwMuS6i8WOzUi7injOs2smrRQsoyIN4CNGhn\/EbBTS6yjbMkyIp7E92yYWRPcU7qZWRbuKd3MLJsqypVOlmaWoyrKlk6WZpYT95RuZpaJ2yzNzEpwT+lmZllVUbZ0sjSz3NRU0XG4k6WZ5aZ6UqWTpZnlxRelm5llVT3Z0snSzHJR31N6tXCyNLPcVFGudLI0s\/y4ZmlmloFvdzQzy6J6cqWTpZnlp4pypZOlmeVD8h08ZmbZVE+udLI0s\/xUUa50sjSz\/FTRUbiTpZnlxT2lm5mVVG23O9bkHYCZWTVwzdLMclNNNUsnSzPLjdsszcxKSC5KzzuK7JwszSw\/TpZmZqX5MNzMLAOf4DEzy6CKcqWTpZnlqIqypZOlmeWmmtosFRF5x7CYpA+Bt\/OOoxFdgVl5B1FlvM2WTqVut74R0a0lC5Q0luTzZjUrIoa2ZAzNUVHJslJJGhcRm+UdRzXxNls63m6Vy\/eGm5ll4GRpZpaBk2U2V+QdQBXyNls63m4Vym2WZmYZuGZpZpaBk6WZWQZOlmZmGThZNkHSQEnfldRWUm3e8VQLb6vmkbSupM0ktcs7FivOJ3gaIWkv4HfAe+kwDrg6Ij7LNbAKJmlARLyWvq6NiLq8Y6p0knYj2c8+At4HzqjfhlZ5XLNsQFJbYB\/gsIjYCbgD6A38SlKnXIOrUOmXfoKkGwAios41zOIkbQVcCBwcETsAnwAn5xuVFeNk2bhOQP\/09e3AXcAKwP5SNfXAV36SVgaOBY4HvpR0HThhZnReRPw7fX0G0MWH45XLybKBiFgIXATsJel7EfEV8CQwAdgm1+AqUETMA34C3ACcCKxYmDDzjK3CPQvcBovbedsBfUl+qJG0Wn6hWWOcLBv3BHA\/cKCkbSOiLiJuAHoBG+UbWuWJiOkRMTciZgFHAu3rE6akTSWtl2+ElSfdp+rbwAXMBj6OiA8ljQTOkdQ+vwitIfdn2YiIWCDpeiCAU9Iv+xdAD2BGrsFVuIj4SNKRwB8kvQLUAjvkHFZFi4hFwFxJ70r6PTAEOCQi5uccmhVwsmxCRHwi6a\/Af0hqSwuAAyJiZr6RVb6ImCXpJWAX4AcRMS3vmCpZ2g7eFvhe+v9OEfF6vlFZQ750KIO0TSnS9ksrQdKqwC3ACRHxUt7xVAtJhwDPR8SkvGOx\/8vJ0spC0ooRsSDvOKqJJIW\/kBXLydLMLAOfDTczy8DJ0swsAydLM7MMnCzNzDJwsmwlJNVJmiBpoqR\/SFrpG5S1vaS70td7SGqygwdJnSUdvRTrOFPSiVnHN5jnakl7N2Nd\/SRNbG6MZoWcLFuP+RGxcURsCHwJ\/LRwohLN\/ntHxJiIOK\/ILJ2BZidLs2rjZNk6PQGsm9aoJku6HHgB6C1piKSnJb2Q1kA7AEgaKukVSU8Ce9UXJOkQSZemr3tIul3Si+mwFXAesE5aq\/1DOt9Jkp6X9JKkswrK+n+SXpX0IDCw1IeQdERazouSbm1QW\/6+pCckvZZ2EYekWkl\/KFj3kd90Q5rVc7JsZSS1IbnN8OV01EDg2ojYBJgHnAZ8PyI2JenU+JeSVgT+CuxOcsvd6k0UfwnwWERsBGwKTCLpg3FqWqs9SdIQku7tNgc2BgZL2lbSYGBfYBOSZPydDB\/ntoj4Trq+ycBhBdP6AdsBuwKj089wGPBpRHwnLf8ISWtlWI9ZSb43vPVoL2lC+voJ4EqSXpLejohn0vFbAoOAp9JuOVcAngbWA96svx857TFoVCPr2BE4CBZ3v\/ZpemtjoSHpUN9PYweS5NkRuD0iPk\/XMSbDZ9pQ0jkkh\/odgPsKpt2S3n76uqQ30s8wBPh2QXvmKum63fu4fWNOlq3H\/IjYuHBEmhDnFY4CHoiI\/RrMtzFJD0stQcDvI+IvDdZx\/FKs42pgz4h4Mb1vevuCaQ3LinTdP4uIwqSKpH7NXK\/Z\/+HD8OXLM8DWktYFkLSSpAHAK8BaktZJ59uvieUfAo5Kl61NH7Mxh6TWWO8+4CcFbaFrSOoOPA78UFJ7SR1JDvlL6QjMUPKoj5ENpo2QVJPGvDbwarruo9L5kTRASU\/uZt+Ya5bLkbRj2UOAG\/X14wtOi4jXJI0C7pY0i6Rn+A0bKeI44ApJhwF1wFER8bSkp9JLc+5N2y3XB55Oa7ZzSbq2e0HSzSQ9zr9N0lRQyukkPYq\/TdIGW5iUXwUeI+lj9KdpH6R\/I2nLfCHt9uxDYM9sW8esOHekYWaWgQ\/DzcwycLI0M8vAydLMLAMnSzOzDJwszcwycLI0M8vAydLMLIP\/D63gDa84r8nkAAAAAElFTkSuQmCC\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre class=\"\">Current Overall accuracy: 0.6521739130434783\r\nTotal Overall Accuracy: 0.6570434782608696\r\n<\/pre>\n<\/div>\n<\/div>\n<div class=\"output_area\">\n<div class=\"output_png output_subarea \"><img src=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+\/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp\/UCwAAIABJREFUeJzt3XmcHFW5xvHfk5lAAknYspCVsAQwoGAIu0oAWQUT1MgmEGQTAeWKYOR6WZQIKheUC4IgymbYJEgMSGQJq2xJ2Akge3YICSEkAZLJe\/+omtCMMz01Q3eqe+b55lOfdFdVn3q7p\/rtU6dOnVJEYGZmxXXIOwAzs2rgZGlmloGTpZlZBk6WZmYZOFmamWXgZGlmlkGbS5aSOkv6u6SFkm7+DOUcKumfpYwtL5K+LOmlz\/D6cyWdXMqYimzrA0kbrYptlUpL9jlJAyWFpNomlp8l6bryRFoakn4g6by841jVckuWkg6RNDn9csyW9A9JXypB0d8CegHrRcTI1hYSEX+JiD1LEE9ZpV+8TYqtExEPRsRmrSy\/B3A48If0+TBJM1pTViNl3yfp6MJ5EdElIl4rRfmlkPH9lmSfq0SSdkn3sXMKZl8OfEdSz7ziykMuyVLSj4DfAr8k2ckGAL8Hhpeg+A2AlyNieQnKqnpN1WBaYBRwR0QsLUE4bVXu+5wSJf0+S+oI\/A54rHB+RHwI\/IPkR7T9iIhVOgFrAR8AI4usszpJMp2VTr8FVk+XDQNmAKcAbwOzgSPTZWcDHwPL0m0cBZwFXFdQ9kAggNr0+SjgNWAR8DpwaMH8hwpetxPwBLAw\/X+ngmX3Ab8AHk7L+SfQvYn3Vh\/\/aQXxjwD2BV4G5gOnF6y\/HfAI8F667sXAaumyB9L3sjh9vwcWlP8TYA5wbf289DUbp9sYkj7vA8wDhjUR773Ad9LHawJLgRXp9j5IX98BGA28CrwL3ASsm76mE3BdOv+99LPrBYwB6oAP03IuTtcPYJP08VXAJcDt6ef6GLBxQWx7Ai+lf5PfA\/cDRzfxPgRcmH7mC4FngC0L9rfzgbeAucBlQOem3m+Dchvb5zoAPwPeTLd3DbBWE\/vfhmnci4C70r9v4f66A\/Cv9LN7uvDvRLLfjSHZ75bWf24l\/K6OBn6d\/h3OabDsUGDSqs4feU6rfoOwN7C8fmdpYp2fA48CPYEe6c7yi3TZsPT1Pwc6kiSZJcA66fKzGuxsDZ+v3FnTL8P7wGbpst7AFunjUaTJElgXWAAclr7u4PT5egU77avApumX7D7gvCbeW338Z6TxHwO8A4wFugJbkCSQjdL1t0m\/MLVp7NOAkwvKW5lcGpT\/K5Ik0JmCZJmuc0xazhrAROD8In+Ld4BtG5Q\/o8E6J6d\/r37pNv8AXJ8uOw74e7qtmvT9dCv43I5uUFbDZDmf5AejFvgLcEO6rHv6t\/tGuuyHJAmrqWS5FzAFWJskcX4O6J0u+y0wPv07d03jPbep99tI2Wfx6X3su8ArwEZAF2AccG3D\/S99\/ghwQfq5fYUkaV6XLutL8iOzL0kC3iN93qPg83sr3WdqgY6NxDaBJNE2Nk0o8p42IPnx7kLjyXIIMH9V5488p1W\/weQXaU4z67wK7NtgR3+jYOddSkGyJfn13qGJHbfh85U7K0myfA\/4JtC5QQyj+CRZHgY83mD5I8Cogp32ZwXLvg\/c2cR7q4+\/Jn3eNY1n+4J1pgAjmnj9ycCtBc8bS5YfA50azGuY4MYDz5LUsFYv8rdYBmzeTFnTgN0LnvdOX1dLkjj+BXyhkbLvo\/lk+ceCZfsCL6aPDwceKVgmYHrD8gqW70by5d8B6NDgdYv5dI11R+D1pt5vI2U33MfuAb5f8Hyzgs+jcP8bQPLDtmbBumP5JFn+hDTJFiyfCBxR8Pn9vEzf09uAAwv+Dg2T5SCgrhzbrtQpjzbLd4HuzbSl9SE5hKn3ZjpvZRnx6fahJSS\/gC0SEYtJDl2\/B8yWdLukzTPEUx9T34Lnc1oQz7sRUZc+rm8LnFuwfGn96yVtKmmCpDmS3idp5+1epGyAdyJpVyrmCmBL4P8i4qMi6y0gSejFbADcKuk9Se+RJM86ksPta0m+4DdImiXp12lbWFZNfa59SJIjAJF8g1eeiJH0fHry8ANJX46Ie0kOcS8B5kq6XFI3kiOXNYApBfHfmc5vrcb231qSz6PhegvS\/bBw3XobACPr40pj+xLJj1G96ZSYpP2BrhFxY5HVupI0Z7QbeSTLR0gOM0cUWWcWyY5Sb0A6rzUWk3wZ6q1fuDAiJkbEHiQ74IskSaS5eOpjmtnKmFriUpK4BkVEN+B0ktpQMVFsoaQuJIeeVwJnSVq3yOrPkDQvFCt7OrBPRKxdMHWKiJkRsSwizo6IwSTtvvvxyYmBonE2YzbJYX\/9e1Lh84jYIpIz610i4sF03kURsQ3JYeumwKkk7bVLSZpf6mNfKyLqk3JrYmxs\/13Op38Q69\/DOpLWbLBuvekkNcvCz3XNiCjsttPc3\/ofBT8aDad\/NPGy3YGh6Q\/0HJIKxcmSbitY53MkbajtxipPlhGxkKS97hJJIyStIamjpH0k\/Tpd7XrgZ5J6SOqert\/avmdPAV+RNEDSWsBP6xdI6iXp6+nO+hFJA31dI2XcAWyadneqlXQgMJikPajcupK0zX2Q1nqPb7B8LknbWEv8DpgSEUeTnDy5rMi6dwC7NNjeeulnWe8yYIykDSDpbiRpePp4V0mfl1STvo9lfPIZtyb2ercDn0\/3oVrgBBr8EBaStK2k7dNa7WKSH+y6iFhB8gN5YX1XGEl9Je1V5P0253rgvyRtmP4w\/RK4scHREBHxJjAZOFvSamnXuf0LVrkO2F\/SXpJqJHVKuzL1I6OI2KfgR6PhtE8TL\/sfkh+TrdNpPMlndGTBOruQnBFvN3LpOhQRFwA\/Ijlj+A7JL+iJwN\/SVc4h2YmeIWlXm5rOa8227gJuTMuawqcTXAeSs+qzSE4k7ELS3tiwjHdJakSnkDQjnAbsFxHzWhNTC\/0YOISk4f8KkvdS6Czg6vQw7dvNFZYmsb1Jmh4g+TsMkXRoEy+5BthXUmeAiHiRJBm8lm6zD0nyHQ\/8U9IikpM926evXx\/4K0minEZy5rf+h+93wLckLZB0UXOxF0o\/+5EkZ2vfJfnxmkzyo9eYbiSf3wKSQ913Sc6AQ9I2+ArwaNrUcTdJO2NT77c5fyJpfniApIfFh8BJTax7CMlnNR84k+Tzrn+P00m6053OJ9+TUynz9zYiFkXEnPqJpOa9OCLmA0jqRNJ+fHU546g0ShtrzZok6ZfA2xHx27xjaUrax3AGSdevSXnH05ZJOgnoHxGn5R3LquRkaVUrPVR+jKTmcyrJofhG4Q70VgZt7tpwa1d2JOlmNo+krW+EE6WVi2uWZmYZuGZpZpbBZx1koaTW6949BmwwMO8wqkpdnY8MWqqDqwgtNv3NN3n33XnN9e9tkZpuG0Qsz95qEkvfmRgRe5cyhpaoqGQ5YIOB3P\/w43mHUVUWLlmWdwhVp1NHZ8uW2mOXHUpeZixfyuqbNdvbbaUPn7qkuSvXyqqikqWZtSeC0o4qV1ZOlmaWDwEq6ZF9WTlZmll+XLM0M2uOoENN3kFk5mRpZvnxYbiZWTOED8PNzJon1yzNzDJxzdLMLAPXLM3MmuNO6WZmzXOndDOzjKqoZlk9kZpZGyOoqck+FStJ6i9pkqRp6W2Qf5jOP0vSTElPpdO+Ba\/5qaRXJL1UcIO6JrlmaWb5KG0\/y+XAKRExVVJXkvvA35UuuzAizi9cWdJg4CCS2yL3Ae6WtGlENHZ3V8A1SzPLk5R9KiIiZkfE1PTxIpI7ifYt8pLhwA0R8VFEvE5yd8\/tim3DydLMcpKeDc86ZS1VGgh8keRmdgAnSnpG0p8krZPO60tya+F6MyieXJ0szSxHLatZdpc0uWA69j+LUxfgFuDkiHgfuBTYGNgamA38b\/2qjURT9LYDbrM0s\/y0rM1yXkQMbbIoqSNJovxLRIwDiIi5BcuvACakT2cA\/Qte3g+YVWzjrlmaWT5aUqtsps1SkoArgWkRcUHB\/N4Fqx0APJc+Hg8cJGl1SRsCg4Ci97RxzdLM8lO6s+E7A4cBz0p6Kp13OnCwpK1JDrHfAI4DiIjnJd0EvEByJv2EYmfCwcnSzPJUoit4IuIhGm+HvKPIa8YAY7Juw8nSzHLia8PNzJonfFsJM7PmuWZpZpaNRx0yM8vANUszswxcszQza4bcZmlmlo1rlmZmzVMVJcvqqQPn5JKLfsv2Qz7PDtt8ge8efggffvhh3iFVnFkzp3Pg8D3ZbYet2H2nL3LlHy4G4Plnn2b4nl9h712242u77cRTU57IOdLK8sPvH8Pgjfryle23XjnvmFGHsOvOQ9l156Fss+Ugdt25yXEjql5yCx5lnvLmZFnErJkzuez3\/8d9Dz\/Oo1Oeoa6ujltuviHvsCpOTU0tP\/v5r7j30ae5beIDXHPlZbz84jR+edbpnHzaf3Pn\/Y9zyk\/P4Jdnn553qBXloEMP54ZxEz4174qrxjLp4clMengyX\/v6AXxt\/xE5RbcKSKhD9ilvPgxvRt3y5SxdupSOHTuydOkS1u\/dJ++QKk6v9XvTa\/1kcJcuXbuyyaDNmTN7JpJYtOh9ABa9v3DlOpbYcecv89abbzS6LCIYf+tfGff3ias2qFWsEmqMWTlZFtGnb19OOvkUttx0IJ06d2a33fdg96\/umXdYFW36W2\/w\/LNP8cVttuPMMedz2Mj9GHPGaFasCG69c1Le4VWNR\/\/1ED169mSjTQblHUpZVVOyLOthuKS90zunvSJpdDm3VQ4LFizg9gnjeWbaq7z02gyWLF7Mjddfl3dYFWvxBx9w3KiDOXPM+XTt1o1r\/3w5Z5zzGx579lXOGPNrTv3B9\/IOsWqM++uNHPCtA\/MOo+zcZglIqgEuAfYBBpOMKze4XNsrh\/vuvZsNBg6ke48edOzYkf1HHMBjjz6Sd1gVadmyZRw36iAO+NZB7JO2s91yw3UrH+83\/Js8PXVyniFWjeXLl3P7+L8x4hsj8w6lvNTCKWflrFluB7wSEa9FxMfADSR3VKsa\/fsPYPLjj7FkyRIigvsn3ctmm30u77AqTkRw6g+OY5NNN+eY7\/9w5fxe6\/fm0YcfAODhByYxcONN8gqxqjww6R4GbboZffr2yzuUshLZa5WVULMsZ5tlY3dP277hSulNh46FJDlVkqHbbc\/wA77JV3YcSm1tLV\/YamtGHXVM3mFVnCce+xfjbhrL5oO3ZO9dkruJnvazn3Peb3\/PWaf\/mLrly1l99U6cd8ElOUdaWY478js8\/NADzH93HlttviGnnX4Ghx5+JLfeclO7OASH6mqzVETRG5q1vmBpJLBXRBydPj8M2C4iTmrqNV\/cZmjc\/3DR22BYAwuXLMs7hKrTqaN7zLXUHrvswFNTp5Q0s9Wut1F02\/eczOsvuO7QKcVuWFZu5axZtvjuaWbWvlRTzbKcP7FPAIMkbShpNeAgkjuqmZlV3QmestUsI2K5pBOBiUAN8KeIeL5c2zOz6iJEhw7V0yRS1k7pEXEHRe6uZmbtWzUdhvsKHjPLT\/XkSidLM8uJXLM0M8vEydLMLAMnSzOzZtRf7lgtnCzNLD\/VkyudLM0sJz7BY2aWjZOlmVkGlXBvnaycLM0sN65Zmpk1o1IG9c3KydLMclNNybJ6hvwwszanVLeVkNRf0iRJ0yQ9L+mH6fx1Jd0l6d\/p\/+uk8yXpovRmis9IGtJcrE6WZpaf0o1nuRw4JSI+B+wAnJDeIHE0cE9EDALuSZ9DciPFQel0LHBpcxtwsjSz3JSqZhkRsyNiavp4ETCN5D5gw4Gr09WuBkakj4cD10TiUWBtSb2LbcNtlmaWjzJ1Spc0EPgi8BjQKyJmQ5JQJfVMV2vshop9gdlNletkaWa5ENDCXNldUuHN5y+PiMs\/VabUBbgFODki3i+SjBtbUPTujU6WZpYT0aFlndLnFbu7o6SOJInyLxExLp09V1LvtFbZG3g7nd\/iGyq6zdLMclPCs+ECrgSmRcQFBYvGA0ekj48AbiuYf3h6VnwHYGH94XpTXLM0s3yoxYfhxewMHAY8K+mpdN7pwHnATZKOAt4CRqbL7gD2BV4BlgBHNrcBJ0szy4WgpYfhTYqIh2i6g9HujawfwAkt2YaTpZnlpoou4HGyNLP8VNPljk6WZpaP0rZZlp2TpZnlIulnWT3Z0snSzHLiIdrMzDKpolzpZGlmOVHpug6tCk6WZpYLt1mamWVURbnSydLM8uOapZlZBlWUKysrWQqoovbeirDp7qfkHULVeXDcmLxDqDofL1tR+kLLNPhvuVRUsjSz9qMVg\/\/mysnSzHLiTulmZplUUa50sjSznLhTuplZ89wp3cwsIydLM7MMqihXOlmaWX5cszQza45HSjcza57cz9LMLJsqypVOlmaWnw5VlC2dLM0sN1WUK50szSwfEtT4Ch4zs+a1iRM8kroVe2FEvF\/6cMysPamiXFm0Zvk8ECSXcNarfx7AgDLGZWZtnEi6D1WLJpNlRPRflYGYWftTRU2WdMiykqSDJJ2ePu4naZvyhmVmbZ6STulZp7w1mywlXQzsChyWzloCXFbOoMysfZCyT3nLcjZ8p4gYIulJgIiYL2m1MsdlZm1ccoPCCsiCGWVJlsskdSA5qYOk9YAy3OrNzNqbKsqVmdosLwFuAXpIOht4CPhVWaMys3ahlG2Wkv4k6W1JzxXMO0vSTElPpdO+Bct+KukVSS9J2qu58putWUbENZKmAF9NZ42MiOeKvcbMrDlluILnKuBi4JoG8y+MiPM\/vW0NBg4CtgD6AHdL2jQi6poqPNPZcKAGWAZ83ILXmJkVpRZMzYmIB4D5GTc9HLghIj6KiNeBV4Dtir0gy9nw\/wauJ8m+\/YCxkn6aMSAzsya18DC8u6TJBdOxGTdzoqRn0sP0ddJ5fYHpBevMSOc1KcsJnu8A20TEkvTNjQGmAOdmDNTM7D8kZ8Nb9JJ5ETG0hZu5FPgFyQnqXwD\/C3yXxiurUaygLMnyzQbr1QKvZQrTzKwpq6CzeUTM\/WRzugKYkD6dARRepdgPmFWsrGIDaVxIkmmXAM9Lmpg+35PkjLiZ2WdS7q5DknpHxOz06QFA\/cnp8SRNiheQNDEOAh4vVlaxmmV9oc8DtxfMf7TFEZuZNaKUNUtJ1wPDSNo2ZwBnAsMkbU1S0XsDOA4gIp6XdBPwArAcOKHYmXAoPpDGlaV4A2ZmjWlFm2VREXFwI7ObzGMRMQYYk7X8ZtssJW2cFjgY6FSwoU2zbqSabbHpRnTp2pWamhpqa2t54F9Fa+rtRr9ea\/PHXxxOr\/W6sSKCP93yMJdcfx\/Xnnckgwb2AmDtrp15b9FSdjjoPAB+\/N09GTV8R+pWrOCUX\/+Vux+ZludbyN2i99\/jnNE\/4NWXpyGJ\/\/nVxWyw0SBOP+lIZs94i979BnDuxVfRba218w61bCphgIysspzguQo4Bzgf2Ac4knZ2uePtE++he\/fueYdRUZbXrWD0BeN46sUZdFljdf419ifc89iLHDb6zyvXOe9HB7Dwg6UAbL7R+ozcawhDvjWG3j3W4o7LTuTzI37OihVFT0C2af\/789HsuMtX+dXvr2HZxx\/z4YdL+PPvL2DbnXZh1PH\/xVWXXsjVl17ISaPPzjvUspCgpoqSZZYO5mtExESAiHg1In5GMgqRtWNz5r3PUy\/OAOCDJR\/x4utz6NPj0zWgb+4xhJvunALAfsO+wM0Tp\/LxsuW8OetdXp0+j223HLiqw64YHyx6nycf\/xfDv50M5tVxtdXo2m1t7r\/rDvb7ZnI0ud83D+a+u24vVkzVq6ZRh7Iky4+U1JVflfQ9SfsDPcscV8WQxIj99ubLO27Ln\/54ed7hVKQBvddl68368cRzb6yct\/OQjZk7fxGvvvUOAH17rMWMOQtWLp\/59gL69FxrVYdaMWZOf4O11+3O2ad9n0P3+zLnjD6JpUsWM3\/e23TvuT4A3Xuuz4J338k50vJqU+NZAv8FdAF+AOwMHEPSqbOoxi5qr0Z3TXqQhx6dzLjbbueKP1zKQw8+kHdIFWXNzqtx\/flHc+r5t7Bo8Ycr539776HcfOfkT1ZsZGeP9nsETt3yOl56\/mm+dehR\/GXCg3RaYw2uuuzCvMNa5dpUzTIiHouIRRHxVkQcFhFfj4iHM5R9FbD3Z44wZ7379AGgR8+e7P\/1EUyZ\/ETOEVWO2toOXH\/+Mdz4j8ncdu\/TK+fX1HRg+G5b8deJU1fOm\/n2e\/Rbf52Vz\/v2XIfZ7yxcpfFWkp69+9Bz\/T5suXVyQcruew\/npeeeYd3uPZn39hwA5r09h3XW65FnmGUlRAdln\/LWZLKUdKukcU1NzRXcwovaK9LixYtZtGjRysf33HMXg7fYIueoKsdlZx7KS6\/P4aLr7v3U\/N2234yX35jLzLffWznv9vueYeReQ1itYy0b9FmPTQb0+NRhe3vTvUcvevXuxxuv\/RuAJ\/51PxsO2oyvfHUfJtxyPQATbrmeXfbYt1gx1a0FtcoKyJVFz4ZfvCoCSC+GPxagf\/\/KumHk23PncsiB3wRg+fLlfPvAg9ljz6qvLJfETltvxKH7bc+zL8\/k0RtGA3DmxeOZ+NALjNxrm5UndupNe20Ot\/zzSZ685b9ZXreCk8+7qV2fCQf48Vm\/4oyTj2HZso\/pO2AgZ\/z696xYsYKfnjiK8TddS68+\/TjvkqvzDrOsKqEtMitFGRuOJA0EJkTEllnWH7LN0HA\/xpbpscMP8g6h6jw4LnM\/ZEsd\/vVhvPDskyXNbD032TIO\/M3Nmde\/+BuDp7RiII2SydLP0sys5ER11SydLM0sN23uvuEAklZvScHpRe2PAJtJmiHpqJYGZ2ZtV\/1tJbJOectybfh2JBejrwUMkLQVcHREnFTsdU1c1G5mtlIF5MDMstQsLwL2A94FiIin8eWOZlYCbaXrUL0OEfFmg4bYouO+mZk1JxmirQKyYEZZkuX09FA8JNUAJwEvlzcsM2sPqulWsVmS5fEkh+IDgLnA3ek8M7PPpIoqls0ny4h4m+Rm5GZmJaMKueY7qyxnw6+gkVtERkTWe\/aamTWqinJlpsPwuwsedyK5Q9r0JtY1M8usmroOZTkMv7HwuaRrgbvKFpGZtQuCiuhsnlVrLnfcENig1IGYWTujNlazlLSAT9osO5CMUTm6nEGZWfsgqidbFk2W6b13tgJmprNWRDnHdDOzdqPU9w0vt6J9QtPEeGtE1KWTE6WZlUwHZZ\/ylqUD\/eOShpQ9EjNrd6rp7o5NHoZLqo2I5cCXgGMkvQosJqk9R0Q4gZpZq1XbYXixNsvHgSHAiFUUi5m1JxUymlBWxZKlACLi1VUUi5m1M23lcscekn7U1MKIuKAM8ZhZO9GWDsNrgC5QRR2hzKyKiJo2UrOcHRE\/X2WRmFm7ktzdMe8osmu2zdLMrCwqpP9kVsWS5e6rLAoza5eq6QRPk53SI2L+qgzEzNqX+sPwUt2wTNKfJL0t6bmCeetKukvSv9P\/10nnS9JFkl6R9EyWC2+q6RYYZtbGdEhHS88yZXAVsHeDeaOBeyJiEHAPnwwCtA8wKJ2OBS5tNtaM78nMrORKWbOMiAdIRkUrNBy4On18NZ9cZDMcuCYSjwJrS+pdrHwnSzPLhUgSUNYJ6C5pcsGU5dY2vSJiNkD6f890fl8+fceHGem8JrVm8F8zs89OtHSAjHkRMbR0W\/8PRUdVc83SzHKjFkytNLf+8Dr9\/+10\/gygf8F6\/YBZxQpysjSzXAiokTJPrTQeOCJ9fARwW8H8w9Oz4jsAC+sP15viw3Azy00pu1lKuh4YRtK2OQM4EzgPuEnSUcBbwMh09TuAfYFXgCXAkc2V72RpZjkp7aC+EXFwE4v+4wKb9K4PJ7SkfCdLM8tF\/dnwauFkaWa5qYTbRWTlZGlmuameVFlhyVJAbU01Vczzd+2fT887hKrz4vxFeYdQdT6sqyt9oS3vZ5mrikqWZtZ+uM3SzCwj1yzNzDJoK4P\/mpmVTXIYXj3Z0snSzHJTRUfhTpZmlhch1yzNzJrnmqWZWTPcZmlmlkXG20VUCidLM8uNk6WZWQY+wWNm1gzhTulmZplkvB94RXCyNLPc+DDczKwZPgw3M8vEV\/CYmTXP\/SzNzLKpolzpZGlm+UjaLKsnXTpZmlluqidVOlmaWZ6qKFs6WZpZbnwYbmaWQfWkSidLM8tTFWVLJ0szy4Xw5Y5mZs1zp3Qzs2yqKFc6WZpZjqooWzpZmllOPJCGmVkmbrNsQ\/458U5+\/KMfUldXx6jvHs2pp43OO6SKdPvYP3LPuLFEBF\/9xiF87dBjWLRwARf+5HjemTWdHn3686NfX0aXbmvnHWrFmHj9lUy6dSwAw0YczN6HHM3FPz2e2W++BsCSRe+zRtdujBk7Mc8wy0aU9ihc0hvAIqAOWB4RQyWtC9wIDATeAL4dEQtaU36H0oTZNtXV1XHyD07gtr\/\/gyefeYGbb7ieaS+8kHdYFeetV17knnFjOffa2zn\/xruY8sDdzH7zNf7250v4\/HZf4v\/GP8znt\/sSf\/vzJXmHWjGmv\/Iik24dy9nXTGDM2Ik89dA9zHnrdU4891LGjJ3ImLET2Xa3fRi66z55h1pWkjJPGe0aEVtHxND0+WjgnogYBNyTPm8VJ8sinnj8cTbeeBM23GgjVlttNUYeeBAT\/n5b3mFVnJmv\/5tBnx\/C6p07U1Nby+BtduDxSXfyxH0TGbb\/SACG7T+SxyfdmXOklWPWG6+wyeeHsHqn5DPbfMj2TC74fCKCx+6ewI57Dc8xyvKTsk+tNBy4On18NTCitQU5WRYxa9ZM+vXrv\/J53779mDlzZo4RVab+G2\/OtKmPsui9+Xy0dClTH7qXeXNmsfDdeazToxcA6\/Toxfvz38050srRb+PNeOnJx1j03gI++nApTz88iflzZ61c\/tKTj7HWut1Zf8CGOUZZfmrBBHSXNLlgOrZBcQH8U9KUgmW9ImI2QPp\/z9bGWrY2S0n9gWuA9YEVwOUR8btyba8cIuI\/5rXgcKDd6LfRIIYlszP+AAAJv0lEQVSPOoFfHH8wnTqvycBNB1NTW5N3WBWt74aD+Nrh3+dXJxxCpzXWYMCgwXSo+eQze2TibezQxmuVrWi0nFdweN2YnSNilqSewF2SXvws4TVUzhM8y4FTImKqpK7AFEl3RUTVNPr17duPGTOmr3w+c+YM+vTpk2NElWv3Aw5m9wMOBmDs\/53Ler16s9Z63VnwzlzW6dGLBe\/Mpdu66+UcZWUZNuIgho04CICbLjmPdXv2BqBu+XImT7qTX1x7R57hrRKl7DoUEbPS\/9+WdCuwHTBXUu+ImC2pN\/B2a8sv22F4RMyOiKnp40XANKBvubZXDkO33ZZXXvk3b7z+Oh9\/\/DE333gDX9vv63mHVZEWzp8HwDuzZ\/LYvf9g571HMHSXPbnv7zcDcN\/fb2bbYXvlGWLFqf\/M5s2ZyeR771zZPvn84w\/Se+DGrNurd57hlZ0oXZulpDXTShmS1gT2BJ4DxgNHpKsdAbT6pMMq6TokaSDwReCxVbG9UqmtreXC313M\/l\/bi7q6Oo4Y9V0Gb7FF3mFVpPN\/fAyL3ltAbW0tR48eQ5dua3PAkSdwwU++x71\/u57uvfvyo1\/\/Ie8wK8pFpx3LBwvfo6a2liN+cg5rpt2qHvnneHbcs40fgqdK2KjVC7g1bSarBcZGxJ2SngBuknQU8BYwsrUbUGPtcqUkqQtwPzAmIsY1svxY4FiA\/gMGbPPyq2+WNZ62ZsJzs5pfyT7lw7oVeYdQdc44bF9ee+GZkjbYb7nVkLj5zgczrz+4T5cpzbRZllVZz4ZL6gjcAvylsUQJEBGXR8TQiBjao3uPcoZjZhVGLfiXt3KeDRdwJTAtIi4o13bMrHp1yD8HZlbOmuXOwGHAbpKeSqd9y7g9M6s2Lexomaey1Swj4iEq4i2aWSXySOlmZll4pHQzs2yqKFc6WZpZjqooWzpZmllOKqNLUFZOlmaWG7dZmpk1o0J6BGXmZGlm+amibOlkaWa56VBFx+FOlmaWm+pJlU6WZpYXd0o3M8uqerKlk6WZ5aJ+pPRq4WRpZrmpolzpZGlm+XHN0swsA1\/uaGaWRfXkSidLM8tPFeVKJ0szy4fkK3jMzLKpnlzpZGlm+amiXOlkaWb5qaKjcCdLM8uLR0o3M2tWtV3u2CHvAMzMqoFrlmaWm2qqWTpZmllu3GZpZtaMpFN63lFk52RpZvlxsjQza54Pw83MMqimEzzuOmRmuVELpmbLkvaW9JKkVySNLnWsTpZmlp8SZUtJNcAlwD7AYOBgSYNLGaqTpZnlRi3414ztgFci4rWI+Bi4ARheylgrqs1y6tQp8zp31Jt5x9GI7sC8vIOoMv7MWqdSP7cNSl3gk1OnTFxjNXVvwUs6SZpc8PzyiLg8fdwXmF6wbAaw\/WeNsVBFJcuI6JF3DI2RNDkihuYdRzXxZ9Y67elzi4i9S1hcY1XPKGH5Pgw3szZhBtC\/4Hk\/YFYpN+BkaWZtwRPAIEkbSloNOAgYX8oNVNRheAW7vPlVrAF\/Zq3jz60VImK5pBOBiUAN8KeIeL6U21BESQ\/rzczaJB+Gm5ll4GRpZpaBk6WZWQZOlk2QtJmkHSV1TC+lsgz8WbWMpE0kDZW0et6xWHE+wdMISd8AfgnMTKfJwFUR8X6ugVUwSZtGxMvp45qIqMs7pkonaT+S\/exdYA5wZv1naJXHNcsGJHUEDgSOiojdgdtIOrueJqlbrsFVqPRL\/5SksQARUecaZnGSdgLOB46IiF2BBUDJR8qx0nGybFw3YFD6+FZgArAacIhUTSPwlZ+kNYETgZOBjyVdB06YGZ0XEU+mj88E1vXheOVysmwgIpYBFwDfkPTliFgBPAQ8BXwp1+AqUEQsBr4LjAV+TDLYwcqEmWdsFe4xYBysbOddnWSwim7pvPXyC80a42TZuAeBfwKHSfpKRNRFxFigD7BVvqFVnoiYFREfRMQ84Digc33ClDRE0ub5Rlh50n2qvg1cwHvA\/Ih4R9KhwDmSOucXoTXkyx0bEREfSvoLyaglP02\/7B8BvYDZuQZX4SLiXUnHAb+R9CLJpWe75hxWRYuI5cAHkqZLOhfYExgVEUtzDs0KOFk2ISIWSLoCeIGktvQh8J2ImJtvZJUvIuZJeoZk1Oo9ImJG3jFVsrQdvCPw5fT\/3SPi3\/lGZQ2561AGaZtSpO2X1gxJ6wA3AadExDN5x1MtJI0Cnij1ABBWGk6WVhaSOkXEh3nHUU0kKfyFrFhOlmZmGfhsuJlZBk6WZmYZOFmamWXgZGlmloGTZRshqU7SU5Kek3SzpDU+Q1nDJE1IH39dUpMDPEhaW9L3W7GNsyT9OOv8ButcJelbLdjWQEnPtTRGs0JOlm3H0ojYOiK2BD4Gvle4UIkW\/70jYnxEnFdklbWBFidLs2rjZNk2PQhsktaopkn6PTAV6C9pT0mPSJqa1kC7AEjaW9KLkh4CvlFfkKRRki5OH\/eSdKukp9NpJ+A8YOO0VvubdL1TJT0h6RlJZxeU9d+SXpJ0N7BZc29C0jFpOU9LuqVBbfmrkh6U9HI6RBySaiT9pmDbx33WD9KsnpNlGyOpluQyw2fTWZsB10TEF4HFwM+Ar0bEEJJBjX8kqRNwBbA\/ySV36zdR\/EXA\/RGxFTAEeJ5kDMZX01rtqZL2JBnebjtga2AbSV+RtA3JvZy\/SJKMt83wdsZFxLbp9qYBRxUsGwjsAnwNuCx9D0cBCyNi27T8YyRtmGE7Zs3yteFtR2dJT6WPHwSuJBkl6c2IeDSdvwMwGHg4HZZzNeARYHPg9frrkdMRg45tZBu7AYfDyuHXFqaXNhbaM53qx2nsQpI8uwK3RsSSdBvjM7ynLSWdQ3Ko34XkntD1bkovP\/23pNfS97An8IWC9sy10m179HH7zJws246lEbF14Yw0IS4unAXcFREHN1hva5IRlkpBwLkR8YcG2zi5Fdu4ChgREU+n100PK1jWsKxIt31SRBQmVSQNbOF2zf6DD8Pbl0eBnSVtAiBpDUmbAi8CG0raOF3v4CZefw9wfPramvQ2G4tIao31JgLfLWgL7SupJ\/AAcICkzpK6khzyN6crMFvJrT4ObbBspKQOacwbAS+l2z4+XR9JmyoZyd3sM3PNsh1JB5YdBVyvT25f8LOIeFnSscDtkuaRjAy\/ZSNF\/BC4XNJRQB1wfEQ8IunhtGvOP9J2y88Bj6Q12w9IhrabKulGkhHn3yRpKmjO\/5CMKP4mSRtsYVJ+CbifZIzR76VjkP6RpC1zajrs2TvAiGyfjllxHkjDzCwDH4abmWXgZGlmloGTpZlZBk6WZmYZOFmamWXgZGlmloGTpZlZBv8PCnwcnrcaEmYAAAAASUVORK5CYII=\" title=\"\" alt=\"\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\"><\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h2 id=\"Evaluating-the-results\"><span class=\"ez-toc-section\" id=\"Evaluating_the_results\"><\/span>Evaluating the results<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">After executing the cross-validation, the\u00a0<b>overall accuracy<\/b>\u00a0is 0.657 (<b>approximately 66%<\/b>).<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>But, sometimes it&#8217;s not enough to use the overall accuracy. Suppose you have a dataset with datapoints divided into two classes, A and B. Suppose that 90% of the datapoints in your dataset belong to class A. Then you can create a dummy classifier that always predicts class A (without using any learning algorithm) and it achieves 90% accuracy! So, for that reason, other metrics like precision, recall or F1 score for each possible class could be helpful in scenarios like this (recall for class B in this example would have been 0, showing a clear problem predicting items of the minority class).<\/p>\n<p>Having said this, we can take a look at the different confusion matrices that were generated in each iteration of our cross-validation process in order to average the important metrics to see how well our model performs for the different categories.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Class_0\"><\/span><u><b>Class 0<\/b><\/u><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ul>\n<li>Average Precision for class\u00a0<i>0<\/i>: (12 + 8 + 9 + 10 + 8)\/ (2 + 4 + 12 + 0 + 6 + 8 + 4 + 7 + 9 + 3 + 5 + 10 + 0 + 5 + 8) = 47\/83 = 0.5663\u00a0<b>(57%)<\/b><\/li>\n<li>Average Recall for class\u00a0<i>0<\/i>: (12 + 8 + 9 + 10 + 8) \/ (12 + 32 + 8 + 8 + 26 + 8 + 9 + 43 + 9 + 10 + 29 + 13 + 8 + 28 + 17) = 48\/260 = 0.1846\u00a0<b><span style=\"color: red;\">(18%)<\/span><\/b><\/li>\n<li>Average F1 score for class\u00a0<i>0<\/i>: 2 x (0.5663 x 0.1846)\/(0.5663 + 0.1846) =\u00a0<b>0.2784<\/b><\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"Class_1\"><\/span><u><b>Class 1<\/b><\/u><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ul>\n<li>Average Precision for class\u00a0<i>1<\/i>: (281 + 289 + 267 + 273 + 270)\/ (98 + 281 + 32 + 69 + 289 + 26 + 108 + 267 + 43 + 83 + 273+ 29 + 90 270 + 28) = 1380\/1986 = 0.6949\u00a0<b>(69%)<\/b><\/li>\n<li>Average Recall for class\u00a0<i>1<\/i>: (281 + 289 + 267 + 273 + 270) \/ (4 + 281 + 48 + 6 + 289 + 72 + 7 + 267 + 46 + 5 + 273 + 63 + 5 + 270 + 60) = 1380\/1696 = 0.8137\u00a0<b>(81%)<\/b><\/li>\n<li>F1 score for class\u00a0<i>1<\/i>: 2 x (0.6949 x 0.8137)\/(0.6949 + 0.8137) =\u00a0<b>0.7496<\/b><\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"Class_2\"><\/span><u><b>Class 2<\/b><\/u><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ul>\n<li>Average Precision for class\u00a0<i>2<\/i>: (90 + 97 + 82 + 96 + 97)\/ (90 + 48 + 8 + 97 + 72 + 8 + 82 + 46 + 9 + 96 + 63 + 13 + 97 + 60 + 17) = 462\/806 = 0.5732\u00a0<b>(57%)<\/b><\/li>\n<li>Average Recall for class\u00a0<i>2<\/i>: (90 + 97 + 82 + 96 + 97) \/ (2 + 98 + 90 + 0 + 69 + 97 + 4 + 108 + 82 + 3 + 83 + 96 + 0 + 90 + 97) = 462\/919 = 0.5027\u00a0<b>(50%)<\/b><\/li>\n<li>Average F1 score for class\u00a0<i>2<\/i>: 2 x (0.5732 x 0.5027)\/(0.5732 + 0.5027) =\u00a0<b>0.5356<\/b><\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h4><span class=\"ez-toc-section\" id=\"About_precision\"><\/span><b>About precision<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Precision for class X represents how good the model (classifier) is when it predicts that a data point belongs to class X.<\/p>\n<p>We can see that the average precision for class 1 it&#8217;s close to 70% that means that 7 out of 10 of the class 1 predictions were predicted to be of class 1, the true value was predicted successfully. The other 30% were &#8216;confused&#8217;, mainly as items of class 2. You can repeat the same reasoning to see how the model performs when predicting class 0 and 2, achieving 57% precision in both classes, where the main confusion is items in class 1 &#8211; medium (in some way this could be expected, small sizes are confused with medium and some big are also confused with medium, but it\u2019s good to have little confusion between small and big).<\/p>\n<h4><span class=\"ez-toc-section\" id=\"About_recall\"><\/span><b>About recall<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Recall for class X represents how good the model (classifier) is at finding data points that belong to class X.<\/p>\n<p>We can observe that the main issue of our model is finding items from class 0 (small). Despite that almost 57% of the model\u2019s predictions for class 0 are accurate, it still fails to give us as many predictions for class 0 as it should. For that reason, the average recall of class 0 is not as good as we&#8217;d like. On the other hand, our model is very good at finding class 1 (medium) items, successfully finding more than 80% of the items in that class. Finally, the recall for class 2 (big) allows us to see that our model is able to successfully find half of the items in class 2 (big), which is better than its recall for class 0 (small) but is not as great as its recall for class 1 (medium).<\/p>\n<h4><span class=\"ez-toc-section\" id=\"About_the_F1_score\"><\/span><b>About the F1 score<\/b><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The F1 score metric (harmonic average of precision and recall) allows us to combine precision and recall in just one metric, both being important to increase its value. So, since the F1 score for class 0 is much lower than the F1 score for class 2, which is lower than the F1 score for class 1, then we can validate that our previous observations are consistent and that our model performs very well at classifying items of class 1 (medium). It is also good at classifying items of class 2 (big) and although the precision at class 0 (small), suggests that we can trust quite well when the model predicts an item is small, we should note that a high percentage of the small items are not being found by our model. This is probably the first point to be improved upon in the future.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div><\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h2 id=\"Building-the-application\"><span class=\"ez-toc-section\" id=\"Building_the_application\"><\/span>Building the application<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Just as an illustrative step, we are going to now build a simple web application. It will be able to search and pick a story to estimate so you can play a bit with the results of this experiment. By using the filter you can search for some specific topics like &#8220;Java&#8221;, &#8220;Windows&#8221; or whatever you want and then by clicking the card&#8217;s icon on the right, you can see the team&#8217;s average prediction and the AI agent prediction. In order to do this, we used one of the models trained during the cross-validation process. Unfortunately, it can only display predictions for stories in the testing set (note that these stories were not seen by the learning algorithm during the training phase), and not for new stories. The reason for this simplification is that we are not hosting the real model with a backend side. I just generated a Javascript model for now that fits a basic AngularJS application, with the code that you can download from this\u00a0<a href=\"https:\/\/github.com\/waljoel\/story-points-prediction\" class=\"external\" rel=\"nofollow\">GitHub repository<\/a>. The javascript model for the stories was built by adding the predictions to the testing_set dataframe in this way:<\/div>\n<div><\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">ts = testing_set[[\"issuekey\", \"title\", \"description\", \"storypoint\"]] #select only the columns to be serialized<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true\">ts[\"prediction\"] = y_pred # add predictions to the dataframe<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<h4 class=\"prompt input_prompt\"><span class=\"ez-toc-section\" id=\"Later_we_used_the_to_json_method_as_follows\"><\/span>Later we used the \u201cto_json\u201d method as follows:<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre class=\"lang:python decode:true \">from IPython.html import widgets\r\njsonDf = ts.to_json(orient='records')\r\nwidgets.HTML(value = ''' backlog_items = ''' + jsonDf)\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">In a real project, a more realistic architecture for hosting, consuming and updating a predictive model should be considered. But for a prototype, I believe this is good enough. So, here you have the prototype! <b>Remember, to see the AI&#8217;s prediction vs the team&#8217;s, click the cards on the right side!\u00a0<\/b><\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[2]:<\/div>\n<div class=\"output_html rendered_html output_subarea output_execute_result\"><iframe loading=\"lazy\" width=\"100%\" height=\"430\" marginwidth=\"0\" marginheight=\"0\" hspace=\"0\" vspace=\"0\" frameborder=\"0\" src=\"https:\/\/waljoel.github.io\/story-points-prediction\/index.html\" data-mce-fragment=\"1\"><\/iframe><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div><\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h2 id=\"Final-thoughts\"><span class=\"ez-toc-section\" id=\"Final_thoughts_on_software_development_estimation\"><\/span>Final thoughts on software development estimation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">Obtaining good estimations has been a big headache historically in the software industry and different techniques have been tried in many kinds of development processes. In this work, we validated the idea of a new kind of estimation strategy based on machine learning by identifying automatically similar tasks in the past and the historical estimations. This is very similar to what is known as\u00a0<b>estimation based on expert judgment<\/b>, with the only difference that the &#8220;expert&#8221; here is an AI software component.<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<h5><span class=\"ez-toc-section\" id=\"Some_of_the_important_things_that_we_can_conclude_from_this_work_are\"><\/span>Some of the important things that we can conclude from this work are:<span class=\"ez-toc-section-end\"><\/span><\/h5>\n<ul>\n<li>Imbalanced datasets can be difficult to deal with in classification problems.<\/li>\n<li>The use of random downsampling did not achieve better results in our concrete problem.<\/li>\n<li>The use of a basic oversampling improved our results a bit. Some more complex oversampling techniques probably could get even better results and it\u2019s something interesting to try out.<\/li>\n<li>The use of general pre-trained vectors for the English language from Wikipedia improved the results a bit, but not considerably.<\/li>\n<li>The use of more domain-specific pre-trained vectors from other open source repositories was better than using general pre-trained vectors, obtaining better results. It&#8217;s better to have few data but with good quality and very representative, than to have tons of data that are not representative of our domain or data with poor quality or noise.<\/li>\n<li>K-fold cross-validation is a helpful approach for model evaluation, mainly when the dataset is not too large as in our case<\/li>\n<li>Overall accuracy is important, but it&#8217;s not the only metric to be considered when evaluating a classifier. Metrics like Precision, Recall and F1 score can be a good option for evaluation metrics in classification problems.<\/li>\n<\/ul>\n<p>We were able to <strong>train a model that could be taken as a base model from where to start and try to optimize it by tuning different parameters that need more experimenting<\/strong>. Other things that could help to improve the performance of our model is to use a smarter approach for oversampling and maybe the possibility of adding more real representative examples for the minority classes, in particular, more examples of small size items.<\/p>\n<p>Also, it&#8217;s worth mentioning that <strong>if we are bad at estimating and the quality of the estimations in our dataset is poor, then the learning algorithm will probably have poor estimations as well<\/strong>. For this reason, something interesting to test would be to use a curated base history of items for training. That is, items that after having been done, the actual size (metric for the effort or how many hours they took) is added to the historical information.<\/p>\n<p>Finally, although the results were not excellent, they were pretty good.<strong> We validated that a machine learning classification approach can be a good option for addressing the common yet difficult issue of estimating how complex a software development task is<\/strong>.<\/p>\n<p>If you improve upon some of the results presented here, we&#8217;d love to know it! Also share your experience if you use some of these ideas to develop your own jira plugin to make your planning meetings more fun \ud83d\ude42<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"About_us\"><\/span><b>About us<\/b><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\"><a href=\"https:\/\/uruit.com\/\">UruIT<\/a> works with US companies, from startups to Fortune 500 enterprises, as nearshore partner for project definition, application design, and technical development. Our Uruguay and Colombia-based teams have worked on over 150 design and development projects up to date.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Are you ready to make the leap in your software development project? <a href=\"https:\/\/uruit.com\/contact\">Tell us about your project<\/a><\/span> <span style=\"font-weight: 400;\">and let\u2019s talk!<\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Machine learning for software development estimation: how to build a smart component that is able to predict the complexity of a software development task.<\/p>\n","protected":false},"author":25,"featured_media":8909,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[290],"tags":[],"_links":{"self":[{"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/posts\/8900"}],"collection":[{"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/comments?post=8900"}],"version-history":[{"count":6,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/posts\/8900\/revisions"}],"predecessor-version":[{"id":11319,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/posts\/8900\/revisions\/11319"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/media\/8909"}],"wp:attachment":[{"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/media?parent=8900"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/categories?post=8900"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/uruit.com\/blog\/wp-json\/wp\/v2\/tags?post=8900"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}