5. Working with Data

Import Methods Overview

RangerIO supports multiple ways to bring your data in, each optimized for different use cases:

Creating Your First Project

What is a Project?

A project in RangerIO is a workspace that contains:
- One or more data sources
- Quality reports and profiles
- AI chat history
- Prepared/cleaned datasets
- Export configurations

Best practices:
- Create separate projects for different initiatives
- Use descriptive project names
- Archive completed projects

Creating a Project

1. Click "New Project" on the home screen
2. Enter project details:
3. Name: Descriptive name (e.g., "Q4 Sales Analysis")
4. Description: Optional notes about the project
5. Location: Where to store project data (default: ~/RangerIO/projects/)
6. Click "Create"

Your new project opens automatically.

Importing Data

Quick Import (Drag & Drop)

The fastest way to import data:

💡 UI Tip: Look for the dashed border "Drop files here" area in the main window

1. Open your project
2. Drag files from your file manager into the RangerIO window
3. Drop over the "Import Data" area
4. RangerIO automatically:
5. Detects file format
6. Infers data types
7. Profiles the data
8. Runs quality checks
9. Enables AI chat

Supported drag & drop:
- Single files
- Multiple files at once
- Entire folders

Detailed Import

For more control over the import process:

1. Click "Import Data" in the left sidebar
2. Choose import method:
3. Upload files
4. Connect to database
5. Scan directory
6. API connection

File Upload

Step-by-step:

1. Click "Upload Files"
2. Select file(s) from browser
3. Configure settings (if needed):
4. CSV/TSV:
   • Delimiter (auto-detected)
   • Has header row? (usually yes)
   • Encoding (UTF-8 default)
   • Date format
5. Excel:
   • Which sheets to import
   • Skip rows (header offset)
6. JSON:
   • Root path (for nested data)
   • Array vs. object mode
7. Click "Import"
8. Wait for processing (progress bar shows status)
9. Review import summary

Database Connection

Connect to external databases for querying:

Supported databases:
- PostgreSQL
- MySQL / MariaDB
- Microsoft SQL Server
- SQLite (local files)

Connection steps:

1. Click "Connect to Database"
2. Enter connection details:
   Host: localhost (or IP address) Port: 5432 (default for PostgreSQL) Database: mydb Username: user Password: ••••••••
3. Test connection (recommended)
4. Choose import mode:
5. Import table - Copy data locally
6. Live query - Query directly (slower, always fresh)
7. Select tables/views to import
8. Configure sampling (optional):
9. Import all rows
10. First N rows only
11. Random sample
12. Import

Security note: Connection credentials are stored encrypted locally.

Directory Watching

Automatically import files as they're added to a folder:

1. Click "Scan Directory"
2. Choose folder to watch
3. Configure filters:
4. File patterns (e.g., *.csv, sales_*.xlsx)
5. Subdirectories (include/exclude)
6. File age (only new files)
7. Set monitoring:
8. One-time scan
9. Watch for changes (continuous)
10. Import existing files (if any)
11. RangerIO monitors the folder and auto-imports new files

Use cases:
- Automated daily reports
- Log file analysis
- Batch processing workflows

API Connections

Import data from REST APIs:

1. Click "API Connection"
2. Enter API details:
   URL: https://api.example.com/data Method: GET Headers: (if authentication required) Authorization: Bearer <token>
3. Test API (sends test request)
4. Configure data extraction:
5. JSON path to array
6. Field mapping
7. Pagination (if applicable)
8. Set refresh schedule:
9. Manual only
10. Every N hours/days
11. Import

Viewing Data

Data Table View

After import, your data appears in a table:

Features:
- Sortable columns - Click header to sort
- Filtering - Search within columns
- Column selection - Show/hide columns
- Pagination - Navigate large datasets
- Export view - Save visible data

Column information:
- Data type indicator (📊 numeric, 📝 text, 📅 date)
- Missing value percentage
- Unique value count
- Click column header for detailed stats

Data Profile

Automatic statistical analysis of your data:

Overview Tab:
- Total rows and columns
- Data size (MB)
- Import date and source
- Last refreshed

Column Stats:
For each column, see:
- Numeric columns:
- Min, max, mean, median
- Standard deviation
- Distribution histogram
- Outliers
- Text columns:
- Most common values
- Average length
- Pattern analysis
- Date columns:
- Date range
- Frequency
- Gaps in timeline

Quality Summary:
- Completeness score (% non-null)
- Uniqueness score (% unique values)
- Validity score (format compliance)
- Overall data quality grade (A-F)

Data Preview

Quick preview:
- First 100 rows shown by default
- Adjust preview size: 100, 500, 1000, all
- Random sample option

Column insights:
- Hover over cell values for full content
- Color-coded by data type
- Missing values highlighted
- Outliers marked

Understanding Data Quality Reports

When data is imported, RangerIO runs comprehensive quality checks.

Quality Dashboard

Access via "Data Quality" tab:

Summary metrics:
- ✅ Pass: Issues found: 0
- ⚠️ Warning: Issues found: 1-10
- ❌ Fail: Issues found: 10+

Categories:
1. Completeness - Missing data
2. Accuracy - Format/range issues
3. Consistency - Cross-field validation
4. Uniqueness - Duplicates

Completeness Checks

What's checked:
- Null/empty values
- Required fields
- Completeness percentage per column

Example issues:
- "Email column is 15% empty (150/1000 rows)"
- "Required field 'customer_id' has 5 null values"

Recommendations:
- Fill with default value
- Remove incomplete rows
- Flag for manual review

Accuracy Checks

What's checked:
- Data type violations
- Format compliance (emails, phones, dates)
- Range validation (min/max bounds)
- Domain validation (allowed values)

Example issues:
- "Invalid email format: 'john@' in row 45"
- "Date 'revenue' has negative values in 12 rows"
- "Phone number format inconsistent: 5 formats detected"

Recommendations:
- Correct format
- Remove invalid values
- Standardize format

Consistency Checks

What's checked:
- Cross-field relationships
- Business rules
- Referential integrity

Example issues:
- "End date before start date in 8 rows"
- "Shipping address missing when status='Shipped'"
- "Product ID not found in products table"

Recommendations:
- Fix relationship
- Add missing data
- Update status

Uniqueness Checks

What's checked:
- Duplicate rows
- Primary key violations
- Fuzzy duplicates (similar but not exact)

Example issues:
- "42 exact duplicate rows found"
- "Customer ID '12345' appears 3 times"
- "3 near-duplicate names: 'John Smith', 'John Smith', 'Jon Smith'"

Recommendations:
- Remove duplicates
- Merge records
- Deduplicate with fuzzy matching

PII Detection

RangerIO automatically scans for sensitive information:

Detected PII types:
- 🔒 Social Security Numbers
- 💳 Credit card numbers
- 📧 Email addresses
- 📱 Phone numbers
- 🌐 IP addresses
- 🏠 Physical addresses
- 🪪 Custom patterns (user-defined)

PII Report shows:
- Column name
- PII type detected
- Number of occurrences
- Sample values (masked)
- Confidence score

Actions available:
- Flag for review
- Mask (XXX-XX-1234)
- Remove column
- Encrypt

Refreshing Data

Manual Refresh

To update an existing data source:

1. Go to "Data Sources"
2. Find your source in the list
3. Click refresh icon (🔄)
4. RangerIO:
5. Re-imports from original source
6. Updates statistics
7. Compares with previous version
8. Shows what changed

Change summary:
- Rows added/removed/updated
- New columns
- Schema changes
- Quality score change

Automatic Refresh

Set up scheduled refreshes:

1. Click source name → Settings
2. Enable "Auto-refresh"
3. Set schedule:
4. Every N hours
5. Daily at specific time
6. Weekly on specific day
7. Save

Notifications:
- Refresh success/failure
- Significant changes detected
- Quality score degradation

Working with Multiple Data Sources

Viewing All Sources

The "Data Sources" panel shows:
- Source name and type
- Row/column count
- Last updated
- Quality score
- Size (MB)

Actions:
- ✏️ Edit connection settings
- 🔄 Refresh data
- 🗑️ Remove source
- 📊 View profile
- 💬 Chat with this source

Switching Between Sources

In chat interface:
- Dropdown menu to select active source(s)
- "All sources" option for cross-source queries
- Tags show which source answered came from

In data view:
- Tabs for each source
- Quick switcher (Ctrl/Cmd + K)

Best Practices

Organizing Data

Use descriptive names:
- ✅ "customer_transactions_2024"
- ❌ "data1"

Tag your sources:
- Add tags: #sales, #customer-data, #Q4
- Filter by tags in source list

Document your data:
- Add descriptions to sources
- Note data origin and freshness
- List known issues or caveats

Performance Tips

For large files (>1GB):
- Import during off-peak hours
- Consider sampling for exploration
- Use database connections instead of file import
- Ensure sufficient RAM

For many files:
- Batch import (drag multiple files)
- Use directory watching
- Consider consolidation

For frequent updates:
- Use database connections (live query mode)
- Set up automatic refresh schedules
- Monitor for schema changes

Security Tips

Sensitive data:
- Review PII detection results
- Mask or remove PII before exporting
- Use encryption for data at rest
- Restrict project folder permissions

Database credentials:
- Use read-only database accounts when possible
- Rotate passwords regularly
- Don't share connection strings