diff --git a/.instructions/INSTRUCTIONS.mdoc b/.instructions/INSTRUCTIONS.mdoc deleted file mode 100644 index 92aeba6..0000000 --- a/.instructions/INSTRUCTIONS.mdoc +++ /dev/null @@ -1,210 +0,0 @@ -{% steps %} -{% step title="Introduction to ML Model Generalization" %} - - -### Introduction - - -Welcome to the "Introduction to ML Model Generalization" lab! -This lab is designed to give you a foundational understanding of generalization in machine learning models, and how to prevent -over or under fitting in models. - - -### Learning Objectives - - -- Review generalization and the importance of not over or under fitting models. -- Practice implementing early learning cutoff and learning rate decay - - -### Prerequisites - - -Familiarity with basic ML principals and key concepts around, learning rates, and model structure. - - -{% /step %} - - -{% step title="Synthetic Data Generation" %} - - -### Synthetic Data Generation -Provided below is a basic function to create some synthetic data for classification. This data will have 2000 samples, each with 20 -features, where 5 features do not affect the outcome of the classification and 15 are directly correlated to the classification. The -options for correct classification will only be between two options. Most importantly, the random state is defined to allow repeatability -of the model generation. -```python -X, y = make_classification(n_samples=2000, - n_features=20, - n_classes=2, - n_informative=15, - n_redundant=5, - random_state=42) -scaler = StandardScaler() -X = scaler.fit_transform(X) -``` - - -### Train/Validation/Test Splits -For splitting the data you will first split into the test and training/validation sets. From there you will split out training/validation -into their separate sets of training and validation, resulting in a data distribution of train (64%), val (16%), test (20%). - - -```python -X_trainval, X_test, y_trainval, y_test = train_test_split(X, y, test_size=0.2, random_state=42) -X_train, X_val, y_train, y_val = train_test_split(X_trainval, y_trainval, test_size=0.2, random_state=42) -``` - - -In practice it is best to define your test dataset before model creation begins and to keep it out of production environments entirely -to ensure there is no data leakage or over fitting of a model. - - - - -{% /step %} - - -{% step title="Model and Generalization Feature Setup" %} - -### Introduction -For this lab you will use a basic feed forward neural network because neural networks allow you to implement additional features -such as learning rate schedulers, and early stopping, that more traditional models such as linear regression do not have. - -### Model Setup -For this -model you will use two dense layers of Relu activation functions, allowing for -more complex patterns to be learned, and ending the model with a sigmoid. -When setting up the model you could also include additional generalization techniques such as drop out, which -selectively turns off a certain percentage of neurons to ensure no single neuron within the neural net -learns to perform a single aspect of prediction. - -**Note:** RELU is used to introduce non linearity into a neural networks learning, and sigmoid is used as a classification function - - -```python -model = tf.keras.Sequential([ - tf.keras.layers.Dense(64, activation='relu', input_shape=(20,)), - tf.keras.layers.Dense(32, activation='relu'), - tf.keras.layers.Dense(1, activation='sigmoid') -]) -``` - - -### Loss function and Model Initialization Parameters -For this lab you will be using the Adam optimizer as Adam is a good starting optimizer for most problems. -Adam takes one parameter which is the starting learning rate, Most -models begin with a learning rate under 0.3 and usually closer to 0.1 at most. Here you also define the loss function -as ```binary_crossentropy``` which is a simple loss function that just compares is the predicted values match -the actual values. -```python -model.compile( - optimizer=tf.keras.optimizers.Adam(learning_rate=0.05), - loss='binary_crossentropy', - metrics=['accuracy'] -) -``` - - -#### Learning Rate Scheduler -For your learning rate scheduler in this lab you will be using "ReduceLROnPlateau" from the keras library which sets the learning rate -decay to plateau at a certain amount, ensuring the model never slows to a halt during training. Below the function parameters are defined: -- ```val_loss``` is the loss applied on the validation training set -- ```factor``` is the factor as which the learning rate will be cut -- ```patience``` is the number of epochs between learning rate reductions -- ```min_lr``` defines the lowest possible value of learning rate -- ``` verbose``` has 3 possible values, 0 which returns nothing, 1 which returns a progress bar, -and 2 which displays the values for each epoch individually as its own line. - -```python -lr_scheduler = tf.keras.callbacks.ReduceLROnPlateau( - monitor='val_loss', # metric to monitor - factor=0.5, # reduce by a factor - patience=2, # wait 2 epochs before reducing LR - min_lr=1e-5, # don't reduce below this - verbose=1 -) -``` - - - - -#### Early Stopping - - -Next in the code you will implement the early stopping aspect of the model. This function will prevent the model from over fitting by -returning to a previous model's parameter in the training if the monitored value does not increase by a specific increment. In your case -The monitored value is the validation loss again. A patience of 3 is defined, state the model has 3 times of not incrementing by a -great enough change before the previous model that did, is selected as the final model. ```min_delta``` defines how much the value -needs to change to be considered a large enough value difference to keep the model. Finally the ```restore_best_weights``` set to true -allows the model to restore to the last model that performed the best, in cases where the models ```min_delta``` was not met. This -functionality is important to ensure the model does not overfit to the training data and keeps some aspect of generalization. - - -```python -early_stop = tf.keras.callbacks.EarlyStopping( - monitor='val_loss', - patience=3, - min_delta=0.01, # minimum change to be considered an improvement - restore_best_weights=True, - verbose=1 -) -``` - - -{% /step %} - - -{% step title="Model training" %} - - -### Model Training - - -Finally onto the model training, you will use the basic fit method and set the validation set in the hyper parameters, this lets the -```val_loss``` correctly be used by the learning rate schedule and the early stopping mechanism. For this case the epochs defaulted -to 100 and the verbose is set as 2, this will ensure you have plenty of epochs to end early and the line by line model training information -can help you better understand the values of ```val_loss``` and how they are changing per epoch. As you run the model pay close attention -to the change in ```val_loss``` and how it correlates to when the model initiates early stopping and rolling back to previous models. -```python -model.fit( - X_train, y_train, - validation_data=(X_val, y_val), - epochs=100, - callbacks=[early_stop, lr_scheduler], # your custom early stopping + LR scheduler - verbose=2 -) -``` - - -{% /step %} - - -{% step title="Evaluating Model Results" %} - - -### Evaluating Model Results - - -The following code provides a basic metric test of your neural network. Depending on the domain of the model different levels of accuracy -are acceptable. It's more important to see a considerable increase in accuracy in predictions compared to existing methods, than it is -to hit a particular threshold of accuracy. Accuracy above 99.5% for validation can be a bit concerning as it may be a sign of over fitting, -and an accuracy below previous methods may be a sign of under fitting. - - -```python -y_pred_probs = model.predict(X_test).flatten() -y_pred = (y_pred_probs >= 0.5).astype(int) - - -print("\n Test Set Evaluation:") -print(classification_report(y_test, y_pred)) -print("Confusion Matrix:") -print(confusion_matrix(y_test, y_pred)) -``` - - -{% /step %} -{% /steps %} - diff --git a/.vscode/settings.json b/.vscode/settings.json deleted file mode 100644 index 5d8e9de..0000000 --- a/.vscode/settings.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "terminal.integrated.fontSize": 15, - "editor.fontSize": 15, - "terminal.integrated.defaultProfile.linux": "bash", - "workbench.colorTheme": "Default Dark Modern", - "workbench.startupEditor": "none", - "files.associations": { - "*.md": "markdoc" - }, - "workspace": { - "view": "readme", - "terminals": [ - { - "name": "Terminal", - "active": false - } - ], - "files": [ - "./lab.ipynb" - ], - } - } \ No newline at end of file