UPDATE: added all notebooks used during research and training

README.md

@@ -85,14 +85,6 @@ sentiment-fused/
     └── ui/                       # User interface components
 ```
 
-### Directory Explanation
-
-- **`model_weights/`**: Contains the actual trained model files (`.pth` files) downloaded from Google Drive at inference time.
-- **`src/models/`**: Contains the Python code for model loading, inference, and prediction logic
-- **`src/utils/`**: Contains preprocessing utilities for audio, vision, and text data
-- **`src/config/`**: Contains centralized configuration settings for the entire application
-- **`src/ui/`**: Contains Streamlit UI components and styling
-
 ## Key Features
 
 - **Real-time Analysis**: Instant sentiment predictions with confidence scores

notebooks/{audio_sentiment_analysis.ipynb → audio_wav2vec2_training.ipynb}

@@ -368,8 +368,28 @@
         "from sklearn.utils.class_weight import compute_class_weight\n",
         "\n",
         "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
-        "print(f\"\\nUsing device: {device}\")\n",
+        "print(f\"\\nUsing device: {device}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1854c1a5"
+      },
+      "source": [
+        "## Data Loading and Preparation\n",
         "\n",
+        "This section downloads the RAVDESS and CREMA-D datasets using `kagglehub`. It then processes the metadata to create a DataFrame with file paths and sentiment labels, using a sentiment mapping to group related emotions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4ce1824e"
+      },
+      "outputs": [],
+      "source": [
         "# --- Download & Process Datasets ---\n",
         "RAVDESS_PATH = kagglehub.dataset_download(\"uwrfkaggler/ravdess-emotional-speech-audio\")\n",
         "CREMA_D_PATH = kagglehub.dataset_download(\"ejlok1/cremad\")\n",
@@ -393,8 +413,28 @@
         "        if sentiment: crema_data.append({\"filepath\": os.path.join(crema_audio_path, filename), \"sentiment\": sentiment})\n",
         "crema_df = pd.DataFrame(crema_data)\n",
         "\n",
-        "combined_df = pd.concat([ravdess_df, crema_df], ignore_index=True).sample(frac=1, random_state=42).reset_index(drop=True)\n",
+        "combined_df = pd.concat([ravdess_df, crema_df], ignore_index=True).sample(frac=1, random_state=42).reset_index(drop=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "86de5576"
+      },
+      "source": [
+        "## Wav2Vec2 Preparation and Dataset Class\n",
         "\n",
+        "This section loads the pre-trained Wav2Vec 2.0 feature extractor and defines the `AudioDataset` class. This class handles loading the audio files, resampling them to the target sampling rate, and processing them using the Wav2Vec 2.0 feature extractor. It also includes a `collate_fn` to handle potential errors during audio processing."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7222e5b9"
+      },
+      "outputs": [],
+      "source": [
         "# =============================================================================\n",
         "# STEP 1: WAV2VEC2 PREPARATION\n",
         "# =============================================================================\n",
@@ -427,8 +467,28 @@
         "    return torch.utils.data.dataloader.default_collate(batch)\n",
         "train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True, collate_fn=collate_fn)\n",
         "val_loader = DataLoader(val_dataset, batch_size=16, shuffle=False, collate_fn=collate_fn)\n",
-        "test_loader = DataLoader(test_dataset, batch_size=16, shuffle=False, collate_fn=collate_fn)\n",
+        "test_loader = DataLoader(test_dataset, batch_size=16, shuffle=False, collate_fn=collate_fn)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "31cb70d3"
+      },
+      "source": [
+        "## Model and Training Setup\n",
         "\n",
+        "This section loads the pre-trained Wav2Vec 2.0 model for audio classification and sets up the loss function and optimizer. It also calculates class weights to handle the imbalanced dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "326b7435"
+      },
+      "outputs": [],
+      "source": [
         "# =============================================================================\n",
         "# STEP 2: MODEL & TRAINING SETUP\n",
         "# =============================================================================\n",
@@ -436,8 +496,28 @@
         "model = AutoModelForAudioClassification.from_pretrained(MODEL_CHECKPOINT, num_labels=NUM_CLASSES).to(device)\n",
         "class_weights_np = compute_class_weight('balanced', classes=np.unique(X_train_df['label']), y=X_train_df['label'])\n",
         "class_weights = torch.tensor(class_weights_np, dtype=torch.float32).to(device)\n",
-        "criterion = nn.CrossEntropyLoss(weight=class_weights)\n",
+        "criterion = nn.CrossEntropyLoss(weight=class_weights)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8aaf7e60"
+      },
+      "source": [
+        "## Two-Stage Fine-Tuning\n",
         "\n",
+        "This section implements a two-stage fine-tuning process for the Wav2Vec 2.0 model. In Stage 1, only the classification head is trained with the base model frozen. In Stage 2, all layers are unfrozen and the entire model is fine-tuned with a lower learning rate and a learning rate scheduler."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6bbccd3c"
+      },
+      "outputs": [],
+      "source": [
         "# =============================================================================\n",
         "# STEP 3: TWO-STAGE FINE-TUNING\n",
         "# =============================================================================\n",
@@ -508,8 +588,28 @@
         "\n",
         "    if avg_val_loss < best_val_loss:\n",
         "        best_val_loss = avg_val_loss\n",
-        "        torch.save(model.state_dict(), 'best_wav2vec2_model_2stage.pth')\n",
+        "        torch.save(model.state_dict(), 'best_wav2vec2_model_2stage.pth')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "74a029a7"
+      },
+      "source": [
+        "## Model Evaluation\n",
         "\n",
+        "This section evaluates the final fine-tuned Wav2Vec 2.0 model on the hold-out test set. It loads the best model state dictionary, performs inference, and then displays the classification report and confusion matrix to assess the model's performance."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2a156c15"
+      },
+      "outputs": [],
+      "source": [
         "# =============================================================================\n",
         "# STEP 4: EVALUATE THE FINAL MODEL\n",
         "# =============================================================================\n",
@@ -540,9 +640,6 @@
   "metadata": {
     "accelerator": "GPU",
     "colab": {
-      "collapsed_sections": [
-        "bubcKFNzLDh_"
-      ],
       "gpuType": "T4",
       "provenance": []
     },