magma-code 0.1.1

# Magma Ingest CLI Tool

A Rust-based CLI tool for ingesting code files into the Magma scanner system. This tool automatically collects source code files from your repository, respects ignore patterns, and packages them for analysis.

## Features

- **Simple CLI Interface**: Easy-to-use command with minimal required parameters
- **Git Integration**: Automatically extracts commit hash, branch name, and author information
- **Smart File Filtering**: Uses both default patterns and custom `.magmaignore` files to exclude unwanted files
- **Efficient Packaging**: Creates compressed zip files with metadata for optimal transfer
- **CI/CD Ready**: Designed for seamless integration into continuous integration pipelines

## Installation

### For External Users (Public Access)

**Quick Install (Recommended):**
```bash
# Install via Rust package manager
cargo install magma-code
```

**Direct Binary Download:**
```bash
# Linux x86_64
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-linux-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# macOS Intel
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# macOS ARM (Apple Silicon)
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-aarch64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# Windows (PowerShell)
Invoke-WebRequest -Uri "https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-windows-x86_64.exe" -OutFile "magma-ingest.exe"
```

---

### For Team Members (Private Repository Access)

> **Note**: This is a private repository. You'll need access to the repository to use these installation methods.

**Unix/Linux/macOS (with repository access):**
```bash
# Option 1: Direct from private repo (requires GitHub authentication)
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  -sSL https://raw.githubusercontent.com/bluemagma-compliance/magma-ingest/main/install.sh | bash

# Option 2: Clone and install locally
git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
./install.sh
```

**Windows (PowerShell, with repository access):**
```powershell
# Option 1: Direct from private repo (requires GitHub authentication)
$headers = @{ Authorization = "token YOUR_GITHUB_TOKEN" }
iwr -Headers $headers -useb https://raw.githubusercontent.com/bluemagma-compliance/magma-ingest/main/install.ps1 | iex

# Option 2: Clone and install locally
git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
.\install.ps1
```

### From crates.io (Recommended for External Users)

```bash
cargo install magma-code
```

### From Source

```bash
git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
cargo build --release
```

The binary will be available at `target/release/magma-ingest`.

### Using Local Path

```bash
cargo install --path .
```

### Verify Installation

```bash
magma-ingest --version
```

## Authentication Setup (For Private Repository)

### GitHub Personal Access Token

To access the private repository installation scripts, you'll need a GitHub Personal Access Token:

1. Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
2. Click "Generate new token (classic)"
3. Select scopes: `repo` (for private repository access)
4. Copy the generated token
5. Use it in the installation commands above by replacing `YOUR_GITHUB_TOKEN`

### Alternative: SSH Key Setup

If you have SSH keys configured with GitHub:

```bash
# Clone via SSH instead
git clone git@github.com:bluemagma-compliance/magma-ingest.git
cd magma-ingest
./install.sh
```

## Usage

### Basic Usage

```bash
magma-ingest --org <organization-id> --key <api-key> --repo-name <repository-name>
```

### Advanced Usage

```bash
magma-ingest \
  --org my-org-123 \
  --key sk-1234567890abcdef \
  --repo-name my-awesome-project \
  --root-path backend \
  --ignore-file custom-ignore.txt \
  --verbose
```

### Parameters

- `--org` (required): Organization ID for the Magma scanner
- `--key` (required): API key for authentication
- `--repo-name` (required): Repository name for identification
- `--root-path` (optional): Root path to scan - use '/' for current directory or specify subdirectory
- `--ignore-file` (optional): Custom ignore file path (defaults to `.magmaignore`)
- `--verbose` (optional): Enable verbose logging

### Configuration

The API base URL is configured in `config.yaml` (embedded in the binary) and defaults to `http://localhost`. The tool automatically:
- Scans the current working directory
- Creates temporary zip files in the system temp directory
- Cleans up temporary files after upload

## Ignore Patterns

The tool uses a `.magmaignore` file to specify which files and directories to exclude from scanning. The syntax is similar to `.gitignore` files and supports glob patterns.

### Default Ignore Patterns

The tool automatically ignores common files and directories:

- Version control directories (`.git`, `.svn`, `.hg`)
- Dependency directories (`node_modules`, `target`, `build`, `__pycache__`)
- IDE files (`.vscode`, `.idea`)
- Binary files (`*.exe`, `*.dll`, `*.so`)
- Archive files (`*.zip`, `*.tar.gz`)
- Media files (`*.jpg`, `*.png`, `*.mp4`)
- Log files (`*.log`)

### Custom Ignore File

Create a `.magmaignore` file in your repository root to specify additional patterns:

```
# Custom ignores
config/secrets.json
*.env
private/**
temp/**

# Large data files
data/**
*.csv

# Documentation
docs/**
*.md
```

## Output

The tool performs the following actions:

1. **Collects Files**: Gathers all non-ignored files from your repository
2. **Creates Package**: Generates a zip file with source files and metadata
3. **Uploads to API**: Sends the package to your Magma ingest endpoint via HTTP POST
4. **Cleans Up**: Optionally removes the local zip file after successful upload (use `--keep-zip` to retain)

### API Request Format

The tool sends a multipart/form-data POST request to `{api-base-url}/ingest/ingest/` with:

- `org_id`: Organization identifier
- `repo_name`: Repository name
- `commit_hash`: Git commit hash
- `branch_name`: Git branch name
- `root_path`: Repository root path
- `file`: Zip file containing source code and metadata

### Metadata Format

```json
{
  "version": "1.0",
  "timestamp": "2024-01-15T10:30:00Z",
  "git": {
    "branch": "main",
    "commit_hash": "abc123...",
    "commit_message": "Add new feature",
    "author_name": "John Doe",
    "author_email": "john@example.com"
  },
  "scan_info": {
    "file_count": 150,
    "tool_version": "0.1.0",
    "tool_name": "magma-ingest"
  }
}
```

## CI/CD Integration

### GitHub Actions

```yaml
name: Code Analysis
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Install magma-ingest
        run: cargo install --git <repository-url>

      - name: Run Magma Ingest
        run: magma-ingest --org ${{ secrets.MAGMA_ORG_ID }} --key ${{ secrets.MAGMA_API_KEY }}
```

### GitLab CI

```yaml
stages:
  - scan

magma_scan:
  stage: scan
  image: rust:latest
  script:
    - cargo install --git <repository-url>
    - magma-ingest --org $MAGMA_ORG_ID --key $MAGMA_API_KEY
  variables:
    MAGMA_ORG_ID: $MAGMA_ORG_ID
    MAGMA_API_KEY: $MAGMA_API_KEY
```

## Error Handling

The tool provides clear error messages for common issues:

- **No Git Repository**: Ensure you're running the tool in a Git repository
- **No Files to Ingest**: Check your ignore patterns if no files are being collected
- **Invalid Patterns**: Verify your `.magmaignore` file syntax
- **Permission Issues**: Ensure the tool has read access to files and write access to output directory

## Logging

Use the `--verbose` flag to enable detailed logging:

```bash
magma-ingest --org my-org --key my-key --verbose
```

This will show:
- Files being processed
- Ignore patterns being applied
- Git information extraction
- Zip file creation progress

## Project Structure

```
Magma-yeeter/
├── src/
│   ├── main.rs          # Main application entry point
│   ├── cli.rs           # Command-line argument parsing
│   ├── git.rs           # Git repository integration
│   ├── ignore.rs        # File ignore pattern management
│   ├── zipper.rs        # File compression and packaging
│   └── error.rs         # Error handling and types
├── examples/
│   ├── github-actions.yml  # GitHub Actions workflow example
│   └── gitlab-ci.yml       # GitLab CI configuration example
├── .magmaignore         # Default ignore patterns
├── Cargo.toml           # Rust project configuration
├── Makefile             # Build automation
├── install.sh           # Unix installation script
├── install.bat          # Windows installation script
└── README.md            # This file
```

## Development

### Building

```bash
# Debug build
cargo build

# Release build
cargo build --release

# Or use make
make build
make release
```

### Testing

```bash
cargo test
```

### Running

```bash
# Development
cargo run -- --org test-org --key test-key

# Release binary
./target/release/magma-ingest --org test-org --key test-key
```

### Using Make

```bash
make help      # Show available targets
make build     # Debug build
make release   # Release build
make install   # Install to ~/.local/bin
make test      # Run tests
make clean     # Clean build artifacts
make demo      # Run with sample parameters
```

## License

MIT License - see LICENSE file for details.