🏴‍☠️ Compress data files for GitHub compatibility

- Replace large CSV files (151MB) with compressed zip (20MB)
- Add automatic data extraction on first run
- Update documentation for zip-based data distribution
- Maintain backward compatibility with existing code
- All tests passing with new data handling

Author: EficodeRjpalt

.gitignore

@@ -49,6 +49,9 @@ ehthumbs.db
 Thumbs.db
 
 # Data files (these be large treasures that shouldn't go in git)
+data/kaggle_so_2023/
+# But keep the zip file for distribution
+!data/kaggle_so_2023_data.zip
 data/*.csv
 data/*.json
 data/*.xlsx

README.md

@@ -56,12 +56,33 @@ This application is designed specifically for **data analysts** who need:
 
 ## 🏴‍☠️ Setup Instructions
 
-### 1. Data Setup (Already Done!)
+### 1. Data Setup
 
-The Stack Overflow 2023 survey data is already available in the `data/kaggle_so_2023/` directory with:
-- `survey_results_public.csv` - Main survey responses
-- `survey_results_schema.csv` - Data schema and column descriptions
-- Additional documentation files
+The Stack Overflow 2023 survey data is provided as a compressed zip file to keep the repository size manageable:
+
+**Option A: Automatic Extraction (Recommended)**
+- The application will automatically extract `data/kaggle_so_2023_data.zip` when first run
+- No manual action needed - just start the server!
+
+**Option B: Manual Extraction**
+```bash
+# Navigate to the data directory
+cd data
+
+# Extract the zip file
+unzip kaggle_so_2023_data.zip
+
+# This creates the kaggle_so_2023/ directory with:
+# - survey_results_public.csv (151MB - main survey responses)
+# - survey_results_schema.csv (data schema and column descriptions)
+# - Additional documentation files
+```
+
+**Data Contents:**
+- `survey_results_public.csv` - Main survey responses (151MB)
+- `survey_results_schema.csv` - Data schema and column descriptions  
+- `so_survey_2023.pdf` - Survey documentation
+- `README_2023.txt` - Additional information
 
 ### 2. Install Dependencies
 

app/data_config.py

@@ -6,6 +6,7 @@
 
 import os
 import pandas as pd
+import zipfile
 from typing import Dict, List, Optional, Any
 from dataclasses import dataclass
 from pathlib import Path
@@ -42,8 +43,27 @@ class DataManager:
     def __init__(self, base_data_path: str):
         self.base_data_path = Path(base_data_path)
         self.data_sources = {}
+        self._ensure_data_extracted()
         self._setup_default_sources()
 
+    def _ensure_data_extracted(self):
+        """
+        Yarr! Make sure the data treasure be extracted from zip if needed
+        """
+        zip_file_path = self.base_data_path / "kaggle_so_2023_data.zip"
+        extract_dir = self.base_data_path / "kaggle_so_2023"
+        
+        # If zip exists but extracted directory doesn't, extract it
+        if zip_file_path.exists() and not extract_dir.exists():
+            print("🏴‍☠️ Ahoy! Extracting data treasure from zip file...")
+            try:
+                with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
+                    zip_ref.extractall(extract_dir)
+                print("⚓ Data successfully extracted, matey!")
+            except Exception as e:
+                print(f"🚨 Blimey! Error extracting data: {e}")
+                raise RuntimeError(f"Failed to extract data from zip file: {e}")
+    
     def _setup_default_sources(self):
         """Set up the default data sources we know about"""