cleanse 0.6.0

Small utility to clean up delimited (TSV/CSV) data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# 🔥 cleanse

A small utility to clean up delimited data to make it consumable by standard unix tools.

## Search words

Clean tsv data.
Clean csv data.

## Overview

Under the hood this uses the `csv` crate to parse data as a CSV, respecting quoting and escaping rules. For each field 
`cleanse` will then try to do the following three things:

1. Inside a field, replace any instances of the `delimiter` character with ` `.
2. Inside a field, replace any instances of the terminator `\n` character with ` `.
3. Inside a field, replace any malformed UTF8 with the utf8 replacment character.

If any changes were made to a field a log entry is made with the record number, field number and changes.


## Example

```bash
$ cat data.tsv | cleanse -o cleansed.tsv -
Aug 18 15:28:02.556  INFO cleanse: Record number 23485, field number 35: [TerminatorReplacement]
Aug 18 15:28:02.724  INFO cleanse: Record number 31036, field number 24: [DelimiterReplacement]
Aug 18 15:28:02.984  INFO cleanse: Record number 44053, field number 35: [TerminatorReplacement]
Aug 18 15:28:03.456  INFO cleanse: Record number 66273, field number 35: [TerminatorReplacement]
Aug 18 15:28:05.149  INFO cleanse: Record number 150669, field number 14: [FixedEncoding]

```