kore_fileformat 1.3.3

# 🚀 Kore — Killer Optimized Record Exchange


![Kore Logo](vscode-kore-viewer/logo.jpg)

[![VS Code Extension](https://img.shields.io/visual-studio-marketplace/v/arunkatherashala.kore-viewer?label=VS%20Code%20Extension&color=0078d4&logo=visualstudiocode)](https://marketplace.visualstudio.com/items?itemName=arunkatherashala.kore-viewer)
[![Crates.io](https://img.shields.io/crates/v/kore_fileformat.svg)](https://crates.io/crates/kore_fileformat)
[![PyPI](https://img.shields.io/pypi/v/kore-fileformat.svg)](https://pypi.org/project/kore-fileformat/)
[![License](https://img.shields.io/badge/license-KUOPL%20v1.1-blue.svg)](KUOPL-LICENSE)
[![Rust 1.70+](https://img.shields.io/badge/rust-1.70%2B-orange.svg)](https://www.rust-lang.org/)

**A high-performance, columnar file format for analytics with cloud storage connectors.**

Kore is a Rust-based columnar file format designed for efficient storage and analysis of structured data. It provides zero external dependencies in the base library with optional cloud connectors for AWS S3, Azure Blob Storage, and Google Cloud Storage.

---

## 🚀 Quick Start (5 Minutes)


### Step 1: Install

```bash
pip install kore-fileformat==1.3.3
```

### Step 2: Import

```python
from kore_fileformat import compress_csv

# Compress a CSV file (that's it!)

original, compressed, ratio = compress_csv("data.csv", "data.kore")
print(f"✅ Compressed {ratio:.1%}")
```

### Step 3: Use It

```python
from kore_fileformat import get_kore_info

# Get file information

info = get_kore_info("data.kore")
print(f"Records: {info['total_records']:,}")
```

---

## 🎨 View .kore Files in VS Code


**NEW**: Install the official **Kore File Viewer** extension for VS Code!

[![Install from VS Code Marketplace](https://img.shields.io/visual-studio-marketplace/v/arunkatherashala.kore-viewer?label=Install%20Extension&color=0078d4)](https://marketplace.visualstudio.com/items?itemName=arunkatherashala.kore-viewer)

### Features:

- ✅ **Auto-detection**: Opens .kore files automatically in VS Code
- ✅ **Table display**: View data in an organized table format
- ✅ **Light/Dark theme**: Automatic theme detection
- ✅ **Database import**: Connect to PostgreSQL, MySQL, SQLite, MongoDB, and more
- ✅ **Multi-format export**: Export to CSV, JSON, Parquet, Apache Arrow

**Install now**: https://marketplace.visualstudio.com/items?itemName=arunkatherashala.kore-viewer

---

## 👀 How to View .kore Files (Other Methods)


### Option 1: CLI (No Coding Required!)

```bash
# See file metadata in terminal

kore inspect data.kore

# Convert to viewable format (Excel, JSON, etc.)

kore convert data.kore data.csv --format csv
# Now open data.csv in Excel, Notepad, Sheets, etc.

```

**📌 Full CLI Guide:** [docs/CLI_GUIDE.md](docs/CLI_GUIDE.md) - All commands, examples, workflows

### Option 2: Notepad/Notepad++ (Windows Only)

**Direct support** - Set up once, then right-click any `.kore` file for:
- View Metadata
- Convert to CSV
- Convert to JSON

**📌 Windows Setup Guide:** [notepad-plugin/SETUP_GUIDE.md](notepad-plugin/SETUP_GUIDE.md) - Integration in 1-10 minutes

### Option 3: Python (Programmatic)

```python
from kore_fileformat import read_kore, get_kore_info

# View metadata

info = get_kore_info("data.kore")
print(f"Columns: {info['columns']}, Records: {info['total_records']:,}")

# Export to DataFrame (view in notebook)

df = read_kore("data.kore")
print(df.head())

# Export to CSV/JSON

df.to_csv("data.csv")
```

**📌 Python API Reference:** [docs/PYTHON_API.md](docs/PYTHON_API.md) - Complete functions and workflows

---

**Note:** `.kore` files are binary format (like `.zip`, `.png`). Use the tools above to view contents. See [notepad-plugin/SETUP_GUIDE.md](notepad-plugin/SETUP_GUIDE.md) for Notepad integration details.

---

## 📚 Documentation


### Getting Started

| Document | For Whom | Time |
|----------|----------|------|
| **[docs/CLI_GUIDE.md](docs/CLI_GUIDE.md)** | Everyone (no coding!) | 10 min |
| **[docs/INSTALLATION.md](docs/INSTALLATION.md)** | Everyone | 5 min |
| **[docs/USER_GUIDE.md](docs/USER_GUIDE.md)** | Python users | 15 min |
| **[docs/EXAMPLES.md](docs/EXAMPLES.md)** | Developers | 20 min |

### Reference

| Document | Coverage |
|----------|----------|
| **[docs/API_REFERENCE.md](docs/API_REFERENCE.md)** | Complete API documentation with examples |
| **[docs/PYTHON_API.md](docs/PYTHON_API.md)** | Python functions, classes, workflows |
| **[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)** | FAQ, common issues, solutions |
| **[notepad-plugin/SETUP_GUIDE.md](notepad-plugin/SETUP_GUIDE.md)** | Notepad/Notepad++ integration for Windows |
| **[WEB_VIEWER_PLAN.md](WEB_VIEWER_PLAN.md)** | Web viewer architecture & roadmap (v1.3.0) |
| **[vscode-kore-viewer/](vscode-kore-viewer/)** | VS Code extension source code |

### Advanced Documentation

| Guide | Purpose |
|---|---|
| **[PYTHON_USER_GUIDE.md](PYTHON_USER_GUIDE.md)** | Cloud integration, advanced features |
| **[DOCKER_EMULATORS_GUIDE.md](DOCKER_EMULATORS_GUIDE.md)** | Docker setup, LocalStack, Azurite testing |
| **[DOCUMENTATION_INDEX.md](DOCUMENTATION_INDEX.md)** | Complete documentation index |
| **[CI_CD_SECRETS_SETUP.md](CI_CD_SECRETS_SETUP.md)** | GitHub Actions, publishing setup |

**👉 New users**: Start with [docs/INSTALLATION.md](docs/INSTALLATION.md) → [docs/USER_GUIDE.md](docs/USER_GUIDE.md)

---

## ✨ Features


### Core Library

- ✅ **Zero External Dependencies**: Lightweight base crate
- ✅ **Columnar Format**: Optimized for analytical queries
- ✅ **Compression**: Built-in data compression
- ✅ **Multi-Platform**: Windows, macOS, Linux, web

### Cloud Connectors

- ✅ **AWS S3**: Full implementation in v1.0.0
- ⏳ **Azure Blob Storage**: Full implementation coming in v1.1.0
- ⏳ **Google Cloud Storage**: Full implementation coming in v1.1.0

### Language Bindings

- ✅ **Python**: PyO3 wheel for Python 3.9-3.12
- ✅ **Java**: JNI bindings and Maven package
- ✅ **JavaScript**: NAPI module for Node.js
- ⏳ **Go**: Coming in v1.2.0

### DevOps & CI/CD

- ✅ **GitHub Actions**: 10 automated jobs for testing and publishing
- ✅ **Docker Support**: Integration tests with emulators
- ✅ **Multi-Registry Publishing**: crates.io, PyPI, Maven Central, npm

---

## 📋 What's Included in v1.0.0


| Component | Status | Details |
|---|---|---|
| **Base Library** | ✅ Production | Columnar format, compression, serialization |
| **S3 Connector** | ✅ Production | Read/write to AWS S3 with LocalStack testing |
| **Azure Connector** | ⏳ Prepared | Stub implementations, full SDK in v1.1.0 |
| **GCS Connector** | ⏳ Prepared | Stub implementations, full SDK in v1.1.0 |
| **Python Bindings** | ✅ Production | Wheel installation, PyPI distribution |
| **Java Bindings** | ✅ Production | JNI library, Maven Central distribution |
| **JavaScript Bindings** | ✅ Production | NAPI addon, npm distribution |
| **Integration Tests** | ✅ Complete | 4 comprehensive tests with emulators |
| **Documentation** | ✅ Complete | 8 guides, 2000+ lines, 50+ examples |

---

## 🎯 Use Cases


### Data Analytics

Process large datasets efficiently with columnar storage:
```python
import kore_fileformat
# Store analytics data in columnar format for fast queries

```

### Cloud Data Lakes

Store data directly in S3, Azure, or GCS:
```python
from kore_fileformat import S3Reader
reader = S3Reader(region='us-east-1')
data = reader.read_file('my-bucket', 'path/to/data.kore')
```

### Multi-Language Projects

Use Kore from Python, Java, or JavaScript in the same project:
```python
# Python: import kore_fileformat

# Java: import com.kore.cloud.S3Reader;

# JS: const kore = require('kore-fileformat');

```

---

## 🏗️ Architecture


### Modular Design

```
kore_fileformat/
├── core/           # Base library (zero dependencies)
├── cloud/          # Cloud connectors (optional)
│   ├── s3/        # AWS S3 (working)
│   ├── azure/     # Azure Blob (v1.1+)
│   └── gcs/       # Google Cloud (v1.1+)
└── bindings/       # Language bindings
    ├── python/    # PyO3 wheel
    ├── java/      # JNI library
    └── napi/      # Node.js addon
```

### Feature Gates

```toml
# Base: zero external dependencies

kore_fileformat = "1.0.0"

# With S3

kore_fileformat = { version = "1.0.0", features = ["s3"] }

# With all cloud (v1.1.0+)

kore_fileformat = { version = "1.0.0", features = ["s3", "azure", "gcs"] }

# With Python bindings

# Use: pip install kore-fileformat

```

---

## 📊 Performance


Kore is designed for analytics workloads:
- **Compression**: 5-10x reduction on typical datasets
- **Query Speed**: Columnar format enables fast aggregations
- **Storage**: 10-50 MB files with millions of rows
- **Cloud**: Direct S3/Azure/GCS integration (no intermediate files)



---

## 🛠️ Installation


### Requirements

- **Rust**: 1.70+ (for building from source)
- **Python**: 3.9-3.12 (for Python wheel)
- **Java**: 17+ (for Java bindings)
- **Node.js**: 14+ (for JavaScript bindings)
- **Docker**: 20.10+ (for testing with emulators)

### From PyPI (Recommended for Python)

```bash
pip install kore-fileformat
```

### From crates.io (Rust)

```bash
cargo add kore_fileformat --features s3
```

### Build from Source

```bash
git clone https://github.com/arunkatherashala/Kore.git
cd Kore
cargo build --release --features s3
```

---

## 🧪 Testing


### Run Unit Tests

```bash
cargo test
```

### Run Integration Tests (requires Docker)

```bash
# Start emulators (LocalStack, Azurite, GCS)

docker-compose up -d

# Run tests

cargo test --features s3,azure,gcs --test integration_tests -- --nocapture

# Stop emulators

docker-compose down
```

See [DOCKER_EMULATORS_GUIDE.md](DOCKER_EMULATORS_GUIDE.md) for detailed setup.

---

## 🚀 Cloud Integration


### AWS S3 (v1.0.0 - Working)

```python
from kore_fileformat import S3Reader

reader = S3Reader(region='us-east-1')
data = reader.read_file('bucket', 'object.kore')
reader.write_file('bucket', 'object.kore', data)
```

### Azure Blob Storage (v1.1.0 - Coming Soon)

```python
from kore_fileformat import AzureBlobReader

reader = AzureBlobReader('account', 'key')
data = reader.read_file('container', 'blob.kore')
```

### Google Cloud Storage (v1.1.0 - Coming Soon)

```python
from kore_fileformat import GcsReader

reader = GcsReader('project-id')
data = reader.read_file('bucket', 'object.kore')
```

---

## 🤝 Contributing


We welcome contributions! Here's how:

1. **Report Issues**: [GitHub Issues](https://github.com/arunkatherashala/Kore/issues)
2. **Discuss Ideas**: [GitHub Discussions](https://github.com/arunkatherashala/Kore/discussions)
3. **Submit PRs**: Fork, branch, code, and create a pull request

See [V1_1_ROADMAP.md](V1_1_ROADMAP.md) for planned features and how to help.

---

## 📅 Roadmap


### v1.0.0 (Current) ✅

- S3 connector with full API
- Python, Java, JavaScript bindings
- Integration tests with emulators
- Complete documentation

### v1.1.0 (Q2 2026)

- Azure Blob Storage full implementation
- Google Cloud Storage full implementation
- Performance optimizations
- Streaming support

### v2.0.0 (Q4 2026)

- Go language bindings
- Multi-region support
- Caching layer
- Advanced compression

See [V1_1_ROADMAP.md](V1_1_ROADMAP.md) for detailed phases and milestones.

---

## 📦 Distribution Channels


### Latest Versions

| Platform | Package | Version | Link |
|---|---|---|---|
| **PyPI** | kore-fileformat | 1.0.0 | [PyPI](https://pypi.org/project/kore-fileformat/) |
| **Crates.io** | kore_fileformat | 1.0.0 | [Crates.io](https://crates.io/crates/kore_fileformat) |
| **Maven** | com.arun.kore:kore-cloud-java | 1.0.0 | Coming v1.1 |
| **npm** | kore-fileformat | 1.0.0 | Coming v1.1 |

---

## 🔒 Security


### Features

- Zero external dependencies in base library
- Optional SDKs are version-pinned and updated regularly
- Integration tests verify cloud connectivity
- GitHub Actions security scanning

### Reporting Security Issues

Please email: **arunkatherashala@gmail.com**

---

## 📞 Support & Community


### Getting Help

- **Documentation**: [DOCUMENTATION_INDEX.md](DOCUMENTATION_INDEX.md)
- **Issues**: [GitHub Issues](https://github.com/arunkatherashala/Kore/issues)
- **Discussions**: [GitHub Discussions](https://github.com/arunkatherashala/Kore/discussions)
- **Email**: arunkatherashala@gmail.com

### Stay Updated

- **GitHub**: Star the repository
- **Releases**: Watch for v1.1.0 announcement
- **Email**: Subscribe to release notifications

---

## 📄 License


Kore is licensed under the **Kore Use-Only Proprietary License (KUOPL) v1.1**.

✅ You can use Kore freely for personal, educational, research, and commercial purposes.
❌ You cannot modify, fork, or redistribute modified versions.

See [KUOPL-LICENSE](KUOPL-LICENSE) for full details.

### Commercial Licensing

For custom licensing, enterprise features, or permission to modify: 
[Contact: arunkatherashala@gmail.com](mailto:arunkatherashala@gmail.com)

---

## 👤 Author


**Sai Arun Kumar Ktherashala**
- Email: arunkatherashala@gmail.com
- GitHub: [@arunkatherashala](https://github.com/arunkatherashala)
- LinkedIn: [Sai Arun Kumar](https://linkedin.com/in/arunkatherashala)

---

## 🎯 What's Next?


### For Users

1. **Read**: [PYTHON_USER_GUIDE.md](PYTHON_USER_GUIDE.md) or [DOCKER_EMULATORS_GUIDE.md](DOCKER_EMULATORS_GUIDE.md)
2. **Install**: `pip install kore-fileformat`
3. **Explore**: Check [DOCUMENTATION_INDEX.md](DOCUMENTATION_INDEX.md) for your role

### For Contributors

1. **Review**: [V1_1_ROADMAP.md](V1_1_ROADMAP.md) for v1.1.0 features
2. **Clone**: `git clone https://github.com/arunkatherashala/Kore.git`
3. **Setup**: Follow [DOCKER_EMULATORS_GUIDE.md](DOCKER_EMULATORS_GUIDE.md)
4. **Code**: Create feature branch and submit PR

### For DevOps

1. **Setup**: [CI_CD_SECRETS_SETUP.md](CI_CD_SECRETS_SETUP.md) for automated publishing
2. **Monitor**: GitHub Actions workflows on each push
3. **Release**: Tag v1.0.1 or v1.1.0 to trigger publishing

---

## ✅ Project Status


| Phase | Status | Delivered |
|---|---|---|
| **Phase 1: Core Library** | ✅ Complete | Base Kore format, compression, serialization |
| **Phase 2: Cloud SDKs** | ✅ Partial | S3 working, Azure/GCS coming v1.1 |
| **Phase 3: Language Bindings** | ✅ Complete | Python, Java, JavaScript production-ready |
| **Phase 4: Integration Tests** | ✅ Complete | 4 comprehensive tests with emulators |
| **Phase 5: CI/CD & Publishing** | ✅ Complete | 10 automated jobs, multi-registry support |
| **Documentation** | ✅ Complete | 8 guides, 2000+ lines, 50+ examples |

---

## 🎉 Thank You!


Thank you for choosing Kore! We're excited to see what you build.

**Questions?** Open an issue or discussion on GitHub.  
**Want to help?** Check [V1_1_ROADMAP.md](V1_1_ROADMAP.md) for features to implement.  
**Found a bug?** Report it on [GitHub Issues](https://github.com/arunkatherashala/Kore/issues).

---

**Latest Release**: v1.0.0  
**Last Updated**: May 14, 2026  
**Status**: Production Ready ✅

🚀 **Let's build amazing data infrastructure together!**