jsonl_logger 70.70.70

# JSONL_LOGGER

Async queue-based structured logging with JSONL output — single background writer thread, zero dependencies beyond Rust stdlib.

## Installation

Add to your `Cargo.toml`:

```toml
[dependencies]
jsonl_logger = "0.2.0"
```

Or use local path:
```toml
jsonl_logger = { path = "path/to/jsonl_logger" }
```

## Quick Start

### Set environment variables in .env
```bash
PROJECT_DIRECTORY=/path/to/your/project
LOGS_LOCAL_TIMEZONE=Asia/Kolkata
LOGGER_FILE_NAME=orders  # Optional: default log file name (default: "LOGS")
```

### Alternatively, set LOGGER_FILE_NAME in your module
```rust
std::env::set_var("LOGGER_FILE_NAME", "orders");
use jsonl_logger::jsonl_logger::{init, log_info};

let logger = init().expect("failed to init logger");
log_info(&logger, "Order placed", Some("orders"), None, None, {
    let mut m = std::collections::HashMap::new();
    m.insert("order_id".to_string(), serde_json::json!(123));
    m
});
```

### LOG FILE NAMING
- **logfile_name**: The logical group for log files. Set via:
  1. `LOGGER_FILE_NAME` env var in .env
  2. `std::env::set_var("LOGGER_FILE_NAME", "...")` in your module
  3. `logfile_name` parameter in function/macro call
  4. Falls back to "LOGS" if none set

- **module_name**: The module name. Passed explicitly via `module_name` parameter. Defaults to "unknown_module" if omitted.

- **source_file**: The actual Rust source filename. Pass explicitly via `source_file` parameter.

### Import and use
```rust
use jsonl_logger::{
    flush_logs, init, send_notification, send_notification_async,
    log_error, log_info, log_metric, log_warn,
};
use std::collections::HashMap;

let logger = init().expect("failed to init logger");

log_info(&logger, "User logged in", Some("auth"), None, None, {
    let mut m = HashMap::new();
    m.insert("user_id".to_string(), serde_json::json!(123));
    m
});

log_warn(&logger, "Rate limit approaching", Some("api"), None, None, {
    let mut m = HashMap::new();
    m.insert("remaining".to_string(), serde_json::json!(10));
    m
});

log_error(&logger, "Payment failed", Some("payments"), None, None, {
    let mut m = HashMap::new();
    m.insert("error_code".to_string(), serde_json::json!(500));
    m
});

log_metric(&logger, "api_latency_ms", 142.5, "ms", Some("api"), None, None, {
    let mut m = HashMap::new();
    m.insert("endpoint".to_string(), serde_json::json!("/checkout"));
    m
});

send_notification(&logger, "Application started", None, None, Some("main"));

send_notification_async(
    logger.clone(),
    "Deployment completed".to_string(),
    None,
    None,
    Some("deploy".to_string()),
).await;

flush_logs(&logger);
```

### All functions support optional parameters
- `logfile_name`: Log file name (auto-detected from LOGGER_FILE_NAME env)
- `module_name`: Source module name (defaults to "unknown_module")
- `source_file`: Actual source filename (pass explicitly for accurate tracking)

### Output
```
{PROJECT_DIRECTORY}/_LOGS_DIRECTORY/{YYYY_MM_DD}/LOGS/{logfile_name}.jsonl
```
Example: `/path/to/logs/_LOGS_DIRECTORY/2026_04_04/LOGS/orders.jsonl`

### JSONL fields
`timestamp`, `timestamp_local`, `level`, `logfile_name`, `module_name`, `source_file`, `message`, `**extra_fields`

## Configuration

### Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `PROJECT_DIRECTORY` | Yes | — | Root directory for log storage |
| `LOGS_LOCAL_TIMEZONE` | Yes | — | Local timezone (e.g. `Asia/Kolkata`, `UTC`) |
| `LOGGER_FILE_NAME` | No | `LOGS` | Default log file name |
| `LOGGER_BUFFER_SIZE` | No | `1000` | Entries per buffer before auto-flush |
| `LOGGER_FLUSH_INTERVAL_MS` | No | `500` | Periodic flush interval (ms) |
| `LOGGER_RETRY_MAX_ATTEMPTS` | No | `3` | Max retry attempts for disk writes |
| `LOGGER_RETRY_BACKOFF_BASE_MS` | No | `100` | Base delay for exponential backoff (ms) |
| `LOGGER_MAX_BUFFER_SIZE` | No | `50000` | Maximum buffer entries before drop |
| `LOGGER_DEBUG_PRINT` | No | `false` | Print retry debug info to stderr |

### Common Timezones

```
Asia:       Asia/Kolkata, Asia/Dubai, Asia/Singapore, Asia/Tokyo
Europe:     Europe/London, Europe/Paris, Europe/Berlin
Americas:   America/New_York, America/Chicago, America/Los_Angeles
UTC:        UTC
```

## Output

### File Structure

```
{PROJECT_DIRECTORY}/_LOGS_DIRECTORY/{YYYY_MM_DD}/LOGS/{logfile_name}.jsonl
```

Example: `/path/to/logs/_LOGS_DIRECTORY/2026_04_04/LOGS/orders.jsonl`

### JSONL Format

Each log entry is a valid JSON object on a single line:

```json
{"timestamp":"2026-04-04T03:30:00.000Z","timestamp_local":"2026-04-04T09:00:00.000+0530","level":"INFO","logfile_name":"orders","module_name":"auth","source_file":"main","message":"User logged in","user_id":123}
```

### Dual-Write Behavior

| Function | Writes To |
|----------|------------|
| `log_info()` | `{module}.jsonl` |
| `log_warn()` | `{module}.jsonl`, `{module}.warn.jsonl` |
| `log_error()` | `{module}.jsonl`, `{module}.errors.jsonl` |
| `log_metric()` | `{module}.metrics.jsonl` |
| `send_notification()` | `main_logger.jsonl` |

## Key Features

1. **Single Background Writer Thread** — `std::sync::mpsc` channel; disk I/O never blocks the caller
2. **Per-Module Log Files** — Each module gets its own JSONL file for independent retention
3. **Dual Timestamps** — UTC for machine parsing, local time for human readability
4. **Retry with Exponential Backoff + Jitter** — 3 attempts with exponential backoff and ±25% jitter
5. **Zero Data Loss** — Buffer capped at 50k entries; oldest dropped on overflow
6. **Hard Flush Guarantee** — `flush_logs()` uses ack pattern, no heuristic sleeps
7. **Explicit Source Tracking** — Pass `source_file` parameter for accurate source attribution
8. **Drop Trait** — Automatic flush when Logger goes out of scope
9. **Explicit Initialization** — `init()` returns `Result`, no side effects on import

## Performance

```
┌───────────────────┬──────────┬──────────┬────────────┬─────────────────────────┐
│ Test              │ Logs     │ Time     │ Throughput │ Status                  │
├───────────────────┼──────────┼──────────┼────────────┼─────────────────────────┤
│ Single-thread     │ 10,000   │ 0.21 sec │ 48,458/sec │ ✅ PASS                 │
│ Multi-thread (10) │ 100,000  │ 0.46 sec │ 219,554/sec│ ✅ PASS                 │
│ Source detection  │ 1,000,000│ 0.05 sec │ 18.5M/sec  │ ✅ 54ns/call            │
│ Source detection  │ 1,000,000│ 0.01 sec │ 91.2M/sec  │ ✅ 11ns/call            │
└───────────────────┴──────────┴──────────┴────────────┴─────────────────────────┘
System: Ubuntu 24.04.4 LTS | AMD Ryzen 7 5800H (16 cores, 13Gi RAM) | Rust 1.94.0
Tested: 2026-04-04 UTC (10 runs average)
```

## API Reference

### init() -> Result<Logger>
Initialize the logger, loading config from environment variables.
- Returns: `Ok(Logger)` on success

### log_info(logger, message, logfile_name, module_name, source_file, extra)
Log an info message with optional structured fields.
- `logger`: `&Logger` handle
- `message`: Log message
- `logfile_name`: Log file name (`Option<&str>`, auto-detected if `None`)
- `module_name`: Module name (`Option<&str>`, defaults to "unknown_module")
- `source_file`: Source filename (`Option<&str>`, falls back to "unknown_source")
- `extra`: `HashMap<String, serde_json::Value>` for structured fields

### log_warn(logger, message, logfile_name, module_name, source_file, extra)
Log a warning message. Writes to both main and `.warn.jsonl` files.
- Same parameters as `log_info()`

### log_error(logger, message, logfile_name, module_name, source_file, extra)
Log an error message. Writes to both main and `.errors.jsonl` files.
- Same parameters as `log_info()`

### log_metric(logger, metric_name, value, unit, logfile_name, module_name, source_file, tags)
Log a metric with METR level.
- `metric_name`: Metric identifier (e.g. "api_latency_ms")
- `value`: Numeric value (f64)
- `unit`: Unit label (e.g. "ms", "bytes")
- `logfile_name`: Log file name (`Option<&str>`, auto-detected if `None`)
- `module_name`: Module name (`Option<&str>`, defaults to "unknown_module")
- `source_file`: Source filename (`Option<&str>`, falls back to "unknown_source")
- `tags`: `HashMap<String, serde_json::Value>` for grouping/filtering

### send_notification(logger, message, logfile_name, module_name, source_file)
Send a notification to the main_logger.jsonl file (blocking).
- `message`: Notification text
- `source_file`: Actual source filename (auto-detected if `None`)

### send_notification_async(logger, message, logfile_name, module_name, source_file)
Send a notification to the main_logger.jsonl file (non-blocking, Tokio).
- Same parameters as `send_notification` (owned types for async)

### flush_logs(logger)
Manually flush all buffered logs to disk and shut down writer thread.

## Error Handling

The logger will return `anyhow::Result<T>`:
- `anyhow::anyhow!("PROJECT_DIRECTORY not set")` if not set
- `anyhow::anyhow!("PROJECT_DIRECTORY path does not exist")` if path doesn't exist
- `anyhow::anyhow!("Directory not writable")` if directory is not writable
- `anyhow::anyhow!("LOGS_LOCAL_TIMEZONE not set")` if timezone not set

Non-primitive types in extra fields are handled by `serde_json::Value` — callers explicitly construct JSON values.

## Retry Policy

Disk writes use `with_file_retry()`:
- **Max attempts**: 3
- **Delay**: Exponential backoff (100ms, 200ms, 400ms) with ±25% jitter
- **Retryable**: All `std::io::Error` during file write
- **On exhaustion**: Lines re-buffered; oldest dropped if buffer exceeds cap

## Test Coverage

```bash
cargo test -p jsonl_logger
```

| Function | Tier | Tests | What is tested |
|----------|------|-------|----------------|
| `log_info()` | 1 | 6 | level, message, auto-detect, empty, special chars, 10k msg, dual-source |
| `log_warn()` | 1 | 6 | level, message, auto-detect, empty, special chars, 10k msg, dual-source |
| `log_error()` | 1 | 8 | level, message, auto-detect, empty, special chars, 10k msg, dual-write, info/warn isolation, dual-source |
| `log_metric()` | 1 | 10 | METR level, float/int value, tags, auto-detect, unit default, zero, negative, metric_name field, audit isolation, dual-source |
| `send_notification()` | 1 | 8 | INFO level, message, module_name routing, source_file, empty msg, emoji/unicode, audit isolation, async variant |
| `resolve_logfile_name()` | 2 | 6 | LOGGER_FILE_NAME priority, filename fallback, .rs/.py stripped |
| `get_source_file_with_fallback()` | 2 | 4 | non-empty, stable cache, env var override, file!() accuracy |
| `get_log_path()` | 2 | 10 | suffix routing, ValueError on missing dirs, path construction, .rs/.py stripped, LOGS subdir |
| `with_file_retry()` | 2 | 5 | success on attempt 1, success on attempt 3, exhaustion, all exception types, max_attempts=1 |
| `flush_buffer()` | 2 | 6 | no-op on empty, cleared after write, re-buffered on exhaustion, buffer cap drop, stderr warning, midnight rollover |
| `writer_worker()` | 2 | 10 | shutdown ack, Entry/Metric/Notification variants, periodic flush, buffer threshold, all buffers drained, Drop safety, clone safety |
| `LoggerConfig` | 2 | 6 | missing PROJECT_DIRECTORY, missing LOGS_LOCAL_TIMEZONE, nonexistent path, defaults, overrides, debug_print case-insensitive |
| `get_timestamps()` | 2 | 4 | returns tuple, UTC ends with Z, ISO-8601 ms precision, real clock |

## Dependencies

```toml
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
chrono = { version = "0.4", features = ["serde"] }
tokio = { version = "1", features = ["full"] }
dotenvy = "0.15"
anyhow = "1.0"
```