magma-code 0.1.1

A CLI tool for ingesting code files into Magma scanner with smart file filtering and Git integration
magma-code-0.1.1 is not a library.

Magma Ingest CLI Tool

A Rust-based CLI tool for ingesting code files into the Magma scanner system. This tool automatically collects source code files from your repository, respects ignore patterns, and packages them for analysis.

Features

  • Simple CLI Interface: Easy-to-use command with minimal required parameters
  • Git Integration: Automatically extracts commit hash, branch name, and author information
  • Smart File Filtering: Uses both default patterns and custom .magmaignore files to exclude unwanted files
  • Efficient Packaging: Creates compressed zip files with metadata for optimal transfer
  • CI/CD Ready: Designed for seamless integration into continuous integration pipelines

Installation

For External Users (Public Access)

Quick Install (Recommended):

# Install via Rust package manager
cargo install magma-code

Direct Binary Download:

# Linux x86_64
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-linux-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# macOS Intel
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# macOS ARM (Apple Silicon)
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-aarch64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# Windows (PowerShell)
Invoke-WebRequest -Uri "https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-windows-x86_64.exe" -OutFile "magma-ingest.exe"

For Team Members (Private Repository Access)

Note: This is a private repository. You'll need access to the repository to use these installation methods.

Unix/Linux/macOS (with repository access):

# Option 1: Direct from private repo (requires GitHub authentication)
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  -sSL https://raw.githubusercontent.com/bluemagma-compliance/magma-ingest/main/install.sh | bash

# Option 2: Clone and install locally
git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
./install.sh

Windows (PowerShell, with repository access):

# Option 1: Direct from private repo (requires GitHub authentication)
$headers = @{ Authorization = "token YOUR_GITHUB_TOKEN" }
iwr -Headers $headers -useb https://raw.githubusercontent.com/bluemagma-compliance/magma-ingest/main/install.ps1 | iex

# Option 2: Clone and install locally
git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
.\install.ps1

From crates.io (Recommended for External Users)

cargo install magma-code

From Source

git clone https://github.com/bluemagma-compliance/magma-ingest.git
cd magma-ingest
cargo build --release

The binary will be available at target/release/magma-ingest.

Using Local Path

cargo install --path .

Verify Installation

magma-ingest --version

Authentication Setup (For Private Repository)

GitHub Personal Access Token

To access the private repository installation scripts, you'll need a GitHub Personal Access Token:

  1. Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
  2. Click "Generate new token (classic)"
  3. Select scopes: repo (for private repository access)
  4. Copy the generated token
  5. Use it in the installation commands above by replacing YOUR_GITHUB_TOKEN

Alternative: SSH Key Setup

If you have SSH keys configured with GitHub:

# Clone via SSH instead
git clone git@github.com:bluemagma-compliance/magma-ingest.git
cd magma-ingest
./install.sh

Usage

Basic Usage

magma-ingest --org <organization-id> --key <api-key> --repo-name <repository-name>

Advanced Usage

magma-ingest \
  --org my-org-123 \
  --key sk-1234567890abcdef \
  --repo-name my-awesome-project \
  --root-path backend \
  --ignore-file custom-ignore.txt \
  --verbose

Parameters

  • --org (required): Organization ID for the Magma scanner
  • --key (required): API key for authentication
  • --repo-name (required): Repository name for identification
  • --root-path (optional): Root path to scan - use '/' for current directory or specify subdirectory
  • --ignore-file (optional): Custom ignore file path (defaults to .magmaignore)
  • --verbose (optional): Enable verbose logging

Configuration

The API base URL is configured in config.yaml (embedded in the binary) and defaults to http://localhost. The tool automatically:

  • Scans the current working directory
  • Creates temporary zip files in the system temp directory
  • Cleans up temporary files after upload

Ignore Patterns

The tool uses a .magmaignore file to specify which files and directories to exclude from scanning. The syntax is similar to .gitignore files and supports glob patterns.

Default Ignore Patterns

The tool automatically ignores common files and directories:

  • Version control directories (.git, .svn, .hg)
  • Dependency directories (node_modules, target, build, __pycache__)
  • IDE files (.vscode, .idea)
  • Binary files (*.exe, *.dll, *.so)
  • Archive files (*.zip, *.tar.gz)
  • Media files (*.jpg, *.png, *.mp4)
  • Log files (*.log)

Custom Ignore File

Create a .magmaignore file in your repository root to specify additional patterns:

# Custom ignores
config/secrets.json
*.env
private/**
temp/**

# Large data files
data/**
*.csv

# Documentation
docs/**
*.md

Output

The tool performs the following actions:

  1. Collects Files: Gathers all non-ignored files from your repository
  2. Creates Package: Generates a zip file with source files and metadata
  3. Uploads to API: Sends the package to your Magma ingest endpoint via HTTP POST
  4. Cleans Up: Optionally removes the local zip file after successful upload (use --keep-zip to retain)

API Request Format

The tool sends a multipart/form-data POST request to {api-base-url}/ingest/ingest/ with:

  • org_id: Organization identifier
  • repo_name: Repository name
  • commit_hash: Git commit hash
  • branch_name: Git branch name
  • root_path: Repository root path
  • file: Zip file containing source code and metadata

Metadata Format

{
  "version": "1.0",
  "timestamp": "2024-01-15T10:30:00Z",
  "git": {
    "branch": "main",
    "commit_hash": "abc123...",
    "commit_message": "Add new feature",
    "author_name": "John Doe",
    "author_email": "john@example.com"
  },
  "scan_info": {
    "file_count": 150,
    "tool_version": "0.1.0",
    "tool_name": "magma-ingest"
  }
}

CI/CD Integration

GitHub Actions

name: Code Analysis
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Install magma-ingest
        run: cargo install --git <repository-url>

      - name: Run Magma Ingest
        run: magma-ingest --org ${{ secrets.MAGMA_ORG_ID }} --key ${{ secrets.MAGMA_API_KEY }}

GitLab CI

stages:
  - scan

magma_scan:
  stage: scan
  image: rust:latest
  script:
    - cargo install --git <repository-url>
    - magma-ingest --org $MAGMA_ORG_ID --key $MAGMA_API_KEY
  variables:
    MAGMA_ORG_ID: $MAGMA_ORG_ID
    MAGMA_API_KEY: $MAGMA_API_KEY

Error Handling

The tool provides clear error messages for common issues:

  • No Git Repository: Ensure you're running the tool in a Git repository
  • No Files to Ingest: Check your ignore patterns if no files are being collected
  • Invalid Patterns: Verify your .magmaignore file syntax
  • Permission Issues: Ensure the tool has read access to files and write access to output directory

Logging

Use the --verbose flag to enable detailed logging:

magma-ingest --org my-org --key my-key --verbose

This will show:

  • Files being processed
  • Ignore patterns being applied
  • Git information extraction
  • Zip file creation progress

Project Structure

Magma-yeeter/
├── src/
│   ├── main.rs          # Main application entry point
│   ├── cli.rs           # Command-line argument parsing
│   ├── git.rs           # Git repository integration
│   ├── ignore.rs        # File ignore pattern management
│   ├── zipper.rs        # File compression and packaging
│   └── error.rs         # Error handling and types
├── examples/
│   ├── github-actions.yml  # GitHub Actions workflow example
│   └── gitlab-ci.yml       # GitLab CI configuration example
├── .magmaignore         # Default ignore patterns
├── Cargo.toml           # Rust project configuration
├── Makefile             # Build automation
├── install.sh           # Unix installation script
├── install.bat          # Windows installation script
└── README.md            # This file

Development

Building

# Debug build
cargo build

# Release build
cargo build --release

# Or use make
make build
make release

Testing

cargo test

Running

# Development
cargo run -- --org test-org --key test-key

# Release binary
./target/release/magma-ingest --org test-org --key test-key

Using Make

make help      # Show available targets
make build     # Debug build
make release   # Release build
make install   # Install to ~/.local/bin
make test      # Run tests
make clean     # Clean build artifacts
make demo      # Run with sample parameters

License

MIT License - see LICENSE file for details.