# NSG CLI
A command-line interface for the [Neuroscience Gateway (NSG)](https://www.nsgportal.org/) BRAIN Initiative API.
## Features
- **Secure credential storage** - Store NSG credentials in `~/.nsg/credentials.json`
- **Job management** - Submit, monitor, and download results from NSG HPC jobs
- **Parallel job fetching** - Fast job listing with concurrent API requests (up to 2x speedup)
- **Beautiful CLI** - Colored output with progress indicators
- **XML API support** - Full support for NSG's REST API (XML-based)
- **Multiple commands** - Login, list, status, submit, and download operations
- **Flexible compilation** - Feature flags for parallel vs sequential processing
## Installation
### From `crates.io`
```bash
cargo install nsg-cli
```
### Build from source
```bash
cd nsg-cli
cargo build --release
```
The binary will be at `target/release/nsg`.
### Install globally
```bash
cargo install --path .
```
This installs the `nsg` binary to `~/.cargo/bin/`.
## Performance
### Parallel Job Fetching
By default, `nsg-cli` uses parallel processing to fetch job details concurrently, significantly improving performance when listing many jobs.
**Benchmark Results (43 jobs):**
- **Parallel (default)**: ~1.75 seconds
- **Sequential**: ~3.47 seconds
- **Speedup**: ~2x faster with parallel processing
### Build Options
```bash
# Default build with parallel support (recommended)
cargo build --release
# Build without parallel support (sequential only)
cargo build --release --no-default-features
# Explicitly enable parallel support
cargo build --release --features parallel
```
### When to Use Sequential
Sequential processing may be preferable when:
- Running on systems with limited CPU resources
- API rate limiting is a concern
- You have very few jobs (< 5)
For most users, the default parallel build provides the best performance.
See [BENCHMARKING.md](BENCHMARKING.md) for detailed benchmarking instructions.
## Quick Start
### 1. Login
First, authenticate with your NSG credentials:
```bash
nsg login
```
You'll be prompted for:
- NSG Username
- NSG Password (hidden input)
- NSG Application Key
Your credentials are stored in `~/.nsg/credentials.json` with secure permissions (0600 on Unix).
**Get NSG credentials at:** https://www.nsgportal.org/
### 2. List Jobs
View all your NSG jobs with detailed information (tool, status, dates):
```bash
nsg list
```
The list command automatically fetches detailed information for each job, including:
- Tool name (e.g., PY_EXPANSE)
- Job status (COMPLETED, RUNNING, etc.)
- Submission date
- Completion date
- Failed status
For additional job messages:
```bash
nsg list --detailed
```
### 3. Check Job Status
Monitor a specific job:
```bash
nsg status <JOB_ID>
```
Examples:
```bash
nsg status NGBW-JOB-PY_EXPANSE-xxxxx
nsg status https://nsgr.sdsc.edu:8443/cipresrest/v1/job/username/NGBW-JOB-PY_EXPANSE-xxxxx
```
### 4. Submit a Job
Submit a new job to NSG:
```bash
nsg submit <ZIP_FILE> --tool <TOOL_NAME>
```
Example:
```bash
nsg submit my_analysis.zip --tool PY_EXPANSE
```
Available tools:
- `PY_EXPANSE` - Python on EXPANSE (default)
- `GPU_PY_EXPANSE` - GPU-accelerated Python
- Other NSG tools as supported
### 5. Download Results
Download results from a completed job:
```bash
nsg download <JOB_ID>
```
Specify output directory:
```bash
nsg download <JOB_ID> --output ./my_results
```
## Commands
### `nsg login`
Authenticate and save credentials.
**Options:**
- `-u, --username <USERNAME>` - NSG username (or prompt)
- `-p, --password <PASSWORD>` - NSG password (or prompt securely)
- `-a, --app-key <APP_KEY>` - NSG application key (or prompt)
- `--no-verify` - Skip connection test
**Example:**
```bash
nsg login --username myuser --app-key MY_APP_KEY
```
### `nsg list`
List all jobs for the authenticated user with detailed information (tool, status, dates).
**Note:** By default, this command fetches detailed status for each job using parallel processing for optimal performance.
**Options:**
- `--detailed` - Show job messages for each job
- `--recent <N>` - Show only the N most recent jobs (default: 20)
- `--limit <N>` - Limit number of jobs to display
- `--all` - Show all jobs (override default 20-job limit)
**Job Information Displayed:**
- Job ID
- Tool name (e.g., PY_EXPANSE)
- Status (COMPLETED, RUNNING, FAILED, etc.)
- Submission timestamp
- Completion timestamp (for terminal jobs)
- Failed status
**Examples:**
```bash
nsg list # Show 20 most recent jobs with details
nsg list --all # Show all jobs with details
nsg list --recent 5 # Show 5 most recent jobs
nsg list --limit 10 # Show first 10 jobs
nsg list --detailed # Show job messages for recent jobs
nsg list --all --detailed # Show all jobs with messages
```
### `nsg status <JOB>`
Check status of a specific job.
**Arguments:**
- `<JOB>` - Job URL or Job ID
**Example:**
```bash
nsg status NGBW-JOB-PY_EXPANSE-xxxxx
```
### `nsg submit <ZIP_FILE>`
Submit a new job to NSG.
**Arguments:**
- `<ZIP_FILE>` - Path to ZIP file containing job data
**Options:**
- `-t, --tool <TOOL>` - NSG tool to use (default: PY_EXPANSE)
- `--no-wait` - Don't wait for job submission confirmation
**Example:**
```bash
nsg submit job_data.zip --tool PY_EXPANSE
```
### `nsg download <JOB>`
Download results from a completed job with real-time progress tracking.
**Arguments:**
- `<JOB>` - Job URL or Job ID
**Options:**
- `-o, --output <DIR>` - Output directory (default: ./nsg_results)
**Features:**
- Real-time progress bar showing download speed and ETA
- File size display in appropriate units (B, KB, MB, GB)
- Automatic file size formatting
**Example:**
```bash
nsg download NGBW-JOB-PY_EXPANSE-xxxxx --output ./results
```
## NSG Job Package Structure
When submitting jobs, NSG expects a specific ZIP structure. For Python jobs:
```
job.zip
└── modeldir/
├── input.py # Main Python script (required)
├── data.edf # Input data files
├── params.json # Configuration
└── ... # Other files
```
The main script should be named `input.py` for PY_EXPANSE tool.
## Configuration
Credentials are stored in: `~/.nsg/credentials.json`
**Format:**
```json
{
"username": "your_username",
"password": "your_password",
"app_key": "your_app_key"
}
```
**Security:**
- On Unix systems, the file permissions are set to `0600` (read/write for owner only)
- Never commit this file to version control
- Keep your credentials secure
## API Documentation
This CLI interfaces with the NSG REST API:
- **Base URL:** `https://nsgr.sdsc.edu:8443/cipresrest/v1`
- **Authentication:** HTTP Basic Auth + `cipres-appkey` header
- **Response format:** XML
## Development
### Project Structure
```
nsg-cli/
├── Cargo.toml
├── src/
│ ├── main.rs # CLI entry point
│ ├── lib.rs # Library exports
│ ├── client.rs # NSG API client
│ ├── config.rs # Credential management
│ ├── models.rs # Data structures & XML parsing
│ └── commands/ # CLI commands
│ ├── mod.rs
│ ├── login.rs
│ ├── list.rs
│ ├── status.rs
│ ├── submit.rs
│ └── download.rs
└── README.md
```
### Dependencies
- **clap** - CLI argument parsing
- **reqwest** - HTTP client
- **quick-xml** - XML parsing
- **serde** - Serialization
- **colored** - Terminal colors
- **indicatif** - Progress bars
- **rpassword** - Secure password input
- **rayon** - Parallel iteration for performance
- **criterion** - Benchmarking framework (dev dependency)
### Building
```bash
cargo build # Debug build
cargo build --release # Release build with parallel support (optimized)
cargo build --release --no-default-features # Sequential build
cargo test # Run tests
cargo check # Type checking only (fast)
cargo bench # Run benchmarks
```
### Benchmarking
Compare parallel vs sequential performance:
```bash
# Benchmark with parallel support (default)
cargo bench --features parallel
# Benchmark without parallel support
cargo bench --no-default-features
# View benchmark reports
open target/criterion/report/index.html
```
For detailed benchmarking instructions, see [BENCHMARKING.md](BENCHMARKING.md).
## Troubleshooting
### Authentication Failed
If login fails:
1. Verify credentials at https://www.nsgportal.org/
2. Check that your NSG account is active
3. Ensure application key is correct
4. Try using `--no-verify` to skip connection test and save credentials anyway
### Job Not Found
If `status` or `download` can't find a job:
1. Verify job ID is correct
2. Use `nsg list` to see all your jobs
3. Try using the full job URL instead of just the ID
### Download Failed
If results download fails:
1. Check job is in COMPLETED stage with `nsg status`
2. Verify job has results available
3. Check output directory permissions
## License
MIT License
## Related Projects
- [DDALAB](https://github.com/sdraeger/DDALAB) - Delay Differential Analysis Laboratory
- [NSG Portal](https://www.nsgportal.org/) - Neuroscience Gateway web interface
## Contributing
This tool was created as part of the DDALAB project. For issues or contributions, please use the DDALAB repository.
## Contact
For NSG-related issues: nsghelp@sdsc.edu
For DDALAB issues: https://github.com/sdraeger/DDALAB/issues