# SIM Analytics Platform with Comprehensive API

## Overview

The SIM Analytics Platform is a comprehensive web application built with Streamlit for managing and analyzing 4G/5G SIM card usage data. The system provides client management, SIM tracking, usage analytics, protocol analysis, and reporting capabilities with role-based authentication and professional PDF report generation. 

**NEW**: The platform now includes a comprehensive REST API with two-tier authentication (Admin/User API keys) that enables external systems to programmatically access all platform features, create clients, manage SIM cards, and retrieve analytics data.

## System Architecture

### Frontend Architecture
- **Framework**: Streamlit web application framework
- **UI Components**: Multi-page dashboard with sidebar navigation
- **Visualization**: Built-in Streamlit charts and Matplotlib integration
- **Report Generation**: ReportLab for professional PDF creation

### Backend Architecture
- **Database**: PostgreSQL with psycopg2 direct connection and SQLAlchemy ORM support
- **Authentication**: Custom role-based authentication system (admin/staff roles)
- **API Layer**: Comprehensive REST API with Flask and two-tier API key authentication
- **Data Processing**: Pandas for data manipulation and analysis
- **Connection Management**: Context managers with retry logic and connection pooling

### Application Structure
- **Main App**: Multiple app variants (`app.py`, `app_postgres.py`, `app_production.py`) for different deployment scenarios
- **Modular Design**: Separate modules for authentication, database management, analytics, and reporting
- **Version Management**: Centralized version control with changelog tracking

## Key Components

### Authentication System (`auth.py`)
- **Purpose**: Manages user authentication and authorization
- **Features**: SHA-256 password hashing, role-based permissions, session management
- **Roles**: Admin (full access) and Staff (limited access)
- **Security**: Password change functionality, session timeout handling

### Database Management (`database.py`)
- **Purpose**: Centralized database operations with connection pooling
- **Features**: SQLAlchemy engine with connection retry logic, context managers
- **Tables**: Clients, SIM numbers, usage records, protocol usage, users
- **Connection**: Robust error handling with SSL support and connection pooling

### Usage Analytics (`usage_analyzer.py`)
- **Purpose**: Generates realistic usage patterns and analyzes consumption data
- **Features**: Multiple usage pattern algorithms (normal, business, weekend-heavy)
- **Data Generation**: Smart daily usage distribution with configurable parameters

### Protocol Analysis (`protocol_analyzer.py`)
- **Purpose**: Analyzes and categorizes network protocol usage on individual SIM basis
- **Categories**: Web browsing, video streaming, social media, gaming, file transfer
- **Features**: Per-SIM protocol analysis with exact usage matching, data validation
- **New Features**: Byte-level precision validation, discrepancy detection, usage period tracking

### Report Generation (`report_generator.py`)
- **Purpose**: Creates professional PDF reports for client distribution
- **Features**: Custom styling, charts, tables, comprehensive usage insights
- **Output**: Client-ready reports with branding and detailed analytics

### Data Population System (`data_population_logger.py`)
- **Purpose**: Admin-only feature for intelligent data population
- **Features**: Gap detection, configurable usage ranges, bulk operations
- **Validation**: Prevents data conflicts, comprehensive error logging

### REST API System (`api.py`, `api_manager.py`)
- **Purpose**: Comprehensive REST API for external system integration
- **Features**: Two-tier authentication (Admin/User API keys), full CRUD operations
- **Endpoints**: Clients, SIM cards, usage data, analytics, SIM-based protocol analysis
- **Security**: Rate limiting, API key management, usage logging and monitoring
- **Documentation**: Interactive API documentation and public documentation route
- **Production Domain**: https://radius.telkom.click/api/v1 with public docs at /api-docs
- **New Features**: Individual SIM protocol analysis with exact usage matching

### API Management Interface (`api_key_management.py`, `api_docs.py`)
- **Purpose**: Admin interface for API key generation and management
- **Features**: API key creation, usage monitoring, rate limit configuration
- **Documentation**: Comprehensive API documentation with examples and integration guides
- **Security**: Secure API key generation with SHA-256 hashing and usage tracking

## Data Flow

### Web Interface Flow
1. **Client Onboarding**: Clients are registered with contact information and service details
2. **SIM Management**: SIM cards are assigned to clients with MSISDN and unique identifiers
3. **Usage Recording**: Daily usage data is captured either manually or through data population
4. **Protocol Analysis**: Usage data is automatically analyzed for protocol distribution
5. **Analytics Processing**: Data is aggregated for trend analysis and reporting
6. **Report Generation**: Professional PDF reports are generated for client delivery

### API Integration Flow
1. **API Key Generation**: Admin users create API keys with appropriate access levels
2. **External System Integration**: Other systems authenticate using API keys
3. **Programmatic Client Creation**: External systems create clients and provision SIM cards
4. **Automated Data Collection**: Usage data can be submitted via API endpoints
5. **Real-time Analytics**: External dashboards pull analytics data via API
6. **Monitoring and Logging**: All API usage is tracked and monitored

## External Dependencies

### Core Dependencies
- **Streamlit**: Web application framework
- **Flask**: REST API framework
- **Flask-CORS**: Cross-origin resource sharing for API
- **PostgreSQL**: Primary database system
- **psycopg2**: PostgreSQL database adapter
- **SQLAlchemy**: ORM and connection management
- **Pandas**: Data manipulation and analysis
- **NumPy**: Numerical computing support

### Visualization and Reporting
- **Matplotlib**: Chart generation for analytics
- **ReportLab**: Professional PDF report creation
- **Plotly** (potential): Interactive visualizations

### Security and Utilities
- **hashlib**: Password hashing for authentication
- **datetime**: Date and time handling
- **json**: Data serialization
- **io**: In-memory file operations

## Deployment Strategy

### Environment Configuration
- **Database URL**: Configured via `DATABASE_URL` environment variable
- **Connection Settings**: SSL mode preference with fallback options
- **Connection Pooling**: SQLAlchemy engine with optimized pool settings

### Application Variants
- **Production** (`app_production.py`): Full-featured version with all modules
- **Postgres** (`app_postgres.py`): PostgreSQL-optimized version
- **Minimal** (`app_minimal.py`): Lightweight version for testing

### Database Schema
- **Clients**: Client information and service details
- **SIM Numbers**: SIM card assignments and identifiers
- **Usage Records**: Daily usage data with date ranges
- **Protocol Usage**: Detailed protocol breakdown per usage record
- **Users**: Authentication and role management
- **API Keys**: API key management with two-tier authentication
- **API Usage Logs**: Comprehensive logging of API usage and monitoring

### Security Considerations
- **Password Hashing**: SHA-256 with salt for user authentication
- **Role-based Access**: Admin and staff roles with different permissions
- **Database Security**: SSL connections with proper timeout handling
- **Session Management**: Streamlit session state for user tracking

## Changelog

```
Changelog:
- August 15, 2025. Version 1.1.3 updated - Added SIM-based protocol analysis with exact usage matching, public API documentation route (/api-docs), production domain configuration (https://radius.telkom.click), and comprehensive API endpoints for individual SIM analysis
- August 14, 2025. Version 1.1.3 released - Comprehensive REST API with two-tier authentication, full CRUD operations, external system integration capabilities, API key management interface, and complete API documentation
- July 06, 2025. Version 1.1.2 released - Enhanced automatic usage generation system with 1-38GB configurable range, monthly resets, and real-time tracking
- July 06, 2025. Implemented automatic usage generation system with 1-38GB daily usage, monthly resets, and automatic SIM provisioning
- July 02, 2025. Initial setup
```

## User Preferences

```
Preferred communication style: Simple, everyday language.
```