srgn 0.4.2

A code surgeon for precise text and code transplantation
Documentation

srgn - a code surgeon

A code surgeon for precise text and code transplantation.

Born a Unicode-capable descendant of tr, srgn adds useful actions, acting within precise, optionally language grammar-aware scopes. It suits use cases where...

  • regex doesn't cut it anymore,
  • editor tools such as Rename all are too specific, and not automatable,
  • precise manipulation, not just matching, is required, and lastly and optionally,
  • Unicode-specific trickery is desired.

Usage

For an "end-to-end" example, consider this Python snippet (more languages are supported):

"""GNU module."""

def GNU_says_moo():
    """The GNU -> say moo -> ✅"""

    GNU = """
      GNU
    """  # the GNU...

    print(GNU + " says moo")  # ...says moo

which with an invocation of

cat gnu.py | srgn --python 'doc-strings' '(?<!The )GNU' 'GNU 🐂 is not Unix' | srgn --symbols

can be manipulated to read

"""GNU 🐂 is not Unix module."""

def GNU_says_moo():
    """The GNU → say moo → ✅"""

    GNU = """
      GNU
    """  # the GNU...

    print(GNU + " says moo")  # ...says moo

which demonstrates:

  • language grammar-aware operation: only Python docstrings were manipulated; virtually impossible to replicate in just regex

    Skip ahead to more such showcases below.

  • advanced regex features such as, in this case, negative lookbehind are supported

  • Unicode is natively handled

  • features such as ASCII symbol replacement are provided

Hence the concept of surgical operation: srgn allows you to be quite precise about the scope of your actions, combining the power of both regular expressions and parsers.

[!NOTE]

Without exception, all code snippets in this README are automatically unit-tested using the actual program binary. What is showcased here is guaranteed to work.

Installation

  • binary
  • cargo install
  • in CI, install-action

Walkthrough

The tool is designed around scopes and actions. Scopes narrow down the parts of the input to process. Actions then perform the processing. Generally, both scopes and actions are composable, so more than one of each may be passed. Both are optional (but taking no action is pointless); specifying no scope implies the entire input is in scope.

At the same time, there is considerable overlap with plain tr: the tool is designed to have close correspondence in the most common use cases, and only go beyond when needed.

Actions

The simplest action is replacement. It is specially accessed (as an argument, not an option) for compatibility with tr, and general ergonomics. All other actions are given as flags, or options should they take a value.

Replacement

For example, simple, single-character replacements work as in tr:

$ echo 'Hello, World!' | srgn 'H' 'J'
Jello, World!

The first argument is the scope (literal H in this case). Anything matched by it is subject to processing (replacement by J, the second argument, in this case). However, there is no direct concept of character classes as in tr. Instead, by default, the scope is a regular expression pattern, so its classes can be used to similar effect:

$ echo 'Hello, World!' | srgn '[a-z]' '_'
H____, W____!

The replacement occurs greedily across the entire match by default (note the UTS character class, reminiscent of tr's [:alnum:]):

$ echo 'ghp_oHn0As3cr3T!!' | srgn 'ghp_[[:alnum:]]+' '*' # A GitHub token
*!!

However, in the presence of capture groups, the individual characters comprising a capture group match are treated individually for processing, allowing a replacement to be repeated:

$ echo 'Hide ghp_th15 and ghp_th4t' | srgn '(ghp_[[:alnum:]]+)' '*'
Hide ******** and ********

Advanced regex features are supported, for example lookarounds:

$ echo 'ghp_oHn0As3cr3T' | srgn '(?<=ghp_)([[:alnum:]]+)' '*'
ghp_***********

Take care in using these safely, as advanced patterns come without certain safety and performance guarantees. If they aren't used, performance is not impacted.

The replacement is not limited to a single character. It can be any string, for example to fix this quote:

$ echo '"Using regex, I now have no issues."' | srgn 'no issues' '2 problems'
"Using regex, I now have 2 problems."

The tool is fully Unicode-aware, with useful support for certain advanced character classes:

$ echo 'Mood: 🙂' | srgn '🙂' '😀'
Mood: 😀
$ echo 'Mood: 🤮🤒🤧🦠 :(' | srgn '\p{Emoji_Presentation}' '😷'
Mood: 😷😷😷😷 :(

Beyond replacement

Seeing how the replacement is merely a static string, its usefulness is limited. This is where tr's secret sauce ordinarily comes into play: using its character classes, which are valid in the second position as well, neatly translating from members of the first to the second. Here, those classes are instead regexes, and only valid in first position (the scope). A regular expression being a state machine, it is impossible to match onto a 'list of characters', which in tr is the second (optional) argument. That concept is out the window, and its flexibility lost.

Instead, the offered actions, all of them fixed, are used. A peek at the most common use cases for tr reveals that the provided set of actions covers virtually all of them! Feel free to file an issue if your use case is not covered.

Onto the next action.

Deletion

Removes whatever is found from the input. Same flag name as in tr.

$ echo 'Hello, World!' | srgn -d '(H|W|!)'
ello, orld

[!NOTE] As the default scope is to match the entire input, it is an error to specify deletion without a scope.

Squeezing

Squeezes repeats of characters matching the scope into single occurrences. Same flag name as in tr.

$ echo 'Helloooo Woooorld!!!' | srgn -s '(o|!)'
Hello World!

If a character class is passed, all members of that class are squeezed into whatever class member was encountered first:

$ echo 'The number is: 3490834' | srgn -s '\d'
The number is: 3

Greediness in matching is not modified, so take care:

$ echo 'Winter is coming... 🌞🌞🌞' | srgn -s '🌞+'
Winter is coming... 🌞🌞🌞

[!NOTE] The pattern matched the entire run of suns, so there's nothing to squeeze. Summer prevails.

Invert greediness if the use case calls for it:

$ echo 'Winter is coming... 🌞🌞🌞' | srgn -s '🌞+?' '☃️'
Winter is coming... ☃️

[!NOTE] Again, as with deletion, specifying squeezing without an explicit scope is an error. Otherwise, the entire input is squeezed.

Character casing

A good chunk of tr usage falls into this category. It's very straightforward.

$ echo 'Hello, World!' | srgn --lower
hello, world!
$ echo 'Hello, World!' | srgn --upper
HELLO, WORLD!
$ echo 'hello, world!' | srgn --titlecase
Hello, World!

Normalization

Decomposes input according to Normalization Form D, and then discards code points of the Mark category (see examples). That roughly means: take fancy character, rip off dangly bits, throw those away.

$ echo 'Naïve jalapeño ärgert mgła' | srgn -d '\P{ASCII}' # Naive approach
Nave jalapeo rgert mga
$ echo 'Naïve jalapeño ärgert mgła' | srgn --normalize # Normalize is smarter
Naive jalapeno argert mgła

Notice how mgła is out of scope for NFD, as it is not decomposable (at least that's what ChatGPT whispers in my ear).

Symbols

This action replaces multi-character, ASCII symbols with appropriate single-code point, native Unicode counterparts.

$ echo '(A --> B) != C --- obviously' | srgn --symbols
(A ⟶ B) ≠ C — obviously

Alternatively, if you're only interested in math, make use of scoping:

$ echo 'A <= B --- More is--obviously--possible' | srgn --symbols '<='
A ≤ B --- More is--obviously--possible

As there is a 1:1 correspondence between an ASCII symbol and its replacement, the effect is reversible[^1]:

$ echo 'A ⇒ B' | srgn --symbols --invert
A => B

There is only a limited set of symbols supported as of right now, but more can be added.

German

This action replaces alternative spellings of German special characters (ae, oe, ue, ss) with their native versions (ä, ö, ü, ß).

$ echo 'Gruess Gott, Poeten und Abenteuergruetze!' | srgn --german
Grüß Gott, Poeten und Abenteuergrütze!

This action is based on a word list.

[!NOTE]

  • empty scope and replacement: the entire input will be processed, and no replacement is performed
  • Poeten remained as-is, instead of being naively and mistakenly converted to Pöten
  • as a (compound) word, Abenteuergrütze is not going to be found in any reasonable word list, but was handled properly nonetheless

On request, replacements may be forced, as is potentially useful for names:

$ echo 'Frau Loetter steht ueber der Mauer.' | srgn --german-naive '(?<=Frau )\w+'
Frau Lötter steht ueber der Mauer.

Through positive lookahead, nothing but the salutation was scoped and therefore changed. Mauer correctly remained as-is, but ueber was not processed. A second pass fixes this:

$ echo 'Frau Loetter steht ueber der Mauer.' | srgn --german-naive '(?<=Frau )\w+' | srgn --german
Frau Lötter steht über der Mauer.

[!NOTE]

Options and flags pertaining to some "parent" are prefixed with their parent's name, and will imply their parent when given, such that the latter does not need to be passed explicitly. That's why --german-naive is named as it is, and --german needn't be passed.

This behavior might change once clap supports subcommand chaining.

Some branches are undecidable for this modest tool, as it operates without language context. For example, both Busse (busses) and Buße (penance) are legal words. By default, replacements are greedily performed if legal (that's the whole point of srgn, after all), but there's a flag for toggling this behavior:

$ echo 'Busse und Geluebte 🙏' | srgn --german
Buße und Gelübte 🙏
$ echo 'Busse 🚌 und Fussgaenger 🚶‍♀️' | srgn --german-prefer-original
Busse 🚌 und Fußgänger 🚶‍♀️

Combining Actions

Most actions are composable, unless it would be nonsensical to do so (like for deletion). Their order of application is fixed, so the order of the flags given has no influence (piping multiple runs is an alternative, if needed). Replacements always occur first. Generally, the CLI is designed to prevent misuse and surprises: it prefers crashing to doing something unexpected. Note that lots of combinations are technically possible, but might yield nonsensical results.

$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu
KOEFFIZIENTEN ≠ BRÜCKEN...

A more narrow scope can be specified, and will apply to all actions equally:

$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu '\b\w{1,8}\b'
Koeffizienten != BRÜCKEN...

The word boundaries are required as otherwise Koeffizienten is matched as Koeffizi and enten. Note how the trailing periods cannot be, for example, squeezed. The required scope of \. would interfere with the given one. Regular piping solves this:

$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu '\b\w{1,8}\b' | srgn -s '\.'
Koeffizienten != BRÜCKEN.

The specially treated replacement action is also composable:

$ echo 'Mooood: 🤮🤒🤧🦠!!!' | srgn -s '\p{Emoji}' '😷'
Mooood: 😷!!!

Emojis are first all replaced, then squeezed. Notice how nothing else is squeezed.

Scopes

Scopes are the second driving concept to srgn. In the default case, the main scope is a regular expression. The actions section showcased this use case in some detail, so it's not repeated here. It is given as a first positional argument.

Language grammar-aware scopes

srgn extends this through premade, language grammar-aware scopes, made possible through the excellent tree-sitter library. It offers a queries feature, which works much like pattern matching against a tree data structure.

srgn comes bundled with a handful of the most useful of these queries. Through its discoverable API (either as a library or via CLI, srgn --help), one can learn of the supported languages and available, premade queries. Each supported language comes with an escape hatch, allowing you to run your own, custom ad-hoc queries. The hatch comes in the form of --lang-query <S EXPRESSION>. It allows you to write small, ad-hoc linters, for example to catch code such as:

if x:
    return left
else:
    return right

which, with an invocation of

cat cond.py | srgn --python-query '(if_statement consequence: (block (return_statement (identifier))) alternative: (else_clause body: (block (return_statement (identifier))))) @cond' --upper

will give (TODO: IMPLEMENT --fail)

IF X:
    RETURN LEFT
ELSE:
    RETURN RIGHT

A couple resources exist for getting started with your own queries:

![NOTE] Language scopes are applied first, so whatever regex aka main scope you pass, it operates on each matched language construct individually.

Showcases
Assigning TODOs (TypeScript)

Perhaps you're using a system of TODO notes in comments:

class TODOApp {
    // TODO app for writing TODO lists
    addTodo(todo: TODO): void {
        // TODO: everything, actually 🤷‍♀️
    }
}

and usually assign people to each note. It's possible to automate assigning yourself to every unassigned note (lucky you!) using

cat todo.ts | srgn --typescript 'comments' 'TODO(?=:)' 'TODO(@poorguy)'

which in this case gives

class TODOApp {
    // TODO app for writing TODO lists
    addTodo(todo: TODO): void {
        // TODO(@poorguy): everything, actually 🤷‍♀️
    }
}

Notice the positive lookahead of (?=:), ensuring an actual TODO note is hit (TODO:). Otherwise, the other TODOs mentioned around the comments would be matched as well.

Converting print calls to proper logging (Python)

Say there's code making liberal use of print:

def print_money():
    """Let's print money 💸."""

    amount = 32
    print("Got here.")

    print_more = lambda s: print(f"Printed {s}")
    print_more(23)  # print the stuff

print_money()
print("Done.")

and a move to logging is desired. That's fully automated by a call of

cat money.py | srgn --python 'function-calls' '^print$' 'logging.info'

yielding

def print_money():
    """Let's print money 💸."""

    amount = 32
    logging.info("Got here.")

    print_more = lambda s: logging.info(f"Printed {s}")
    print_more(23)  # print the stuff

print_money()
logging.info("Done.")

[!NOTE] Note the anchors: print_more is a function call as well, but ^print$ ensures it's not matched.

The regular expression applies after grammar scoping, so operates entirely within the already-scoped context.

Remove all comments (C#)

Overdone, comments can turn into smells. If not tended to, they might very well start lying:

using System.Linq;

public class UserService
{
    private readonly AppDbContext _dbContext;

    /// <summary>
    /// Initializes a new instance of the <see cref="FileService"/> class.
    /// </summary>
    /// <param name="dbContext">The configuration for manipulating text.</param>
    public UserService(AppDbContext dbContext)
    {
        _dbContext /* the logging context */ = dbContext;
    }

    /// <summary>
    /// Uploads a file to the server.
    /// </summary>
    // Method to log users out of the system
    public void DoWork()
    {
        _dbContext.Database.EnsureCreated(); // Ensure the database schema is deleted

        _dbContext.Users.Add(new User /* the car */ { Name = "Alice" });

        /* Begin reading file */
        _dbContext.SaveChanges();

        var user = _dbContext.Users.Where(/* fetch products */ u => u.Name == "Alice").FirstOrDefault();

        /// Delete all records before proceeding
        if (user /* the product */ != null)
        {
            System.Console.WriteLine($"Found user with ID: {user.Id}");
        }
    }
}

So, should you count purging comments among your fetishes, more power to you:

cat UserService.cs | srgn --csharp 'comments' -d '.*' | srgn -d '[[:blank:]]+\n'

The result is a tidy, yet taciturn:

using System.Linq;

public class UserService
{
    private readonly AppDbContext _dbContext;

    public UserService(AppDbContext dbContext)
    {
        _dbContext  = dbContext;
    }

    public void DoWork()
    {
        _dbContext.Database.EnsureCreated();
        _dbContext.Users.Add(new User  { Name = "Alice" });

        _dbContext.SaveChanges();

        var user = _dbContext.Users.Where( u => u.Name == "Alice").FirstOrDefault();

        if (user  != null)
        {
            System.Console.WriteLine($"Found user with ID: {user.Id}");
        }
    }
}

Note how all different sorts of comments were identified and removed. The second pass removes all leftover dangling lines ([:blank:] is tabs and spaces).

[!NOTE] When deleting (-d), for reasons of safety and sanity, a scope is required.

Explicit failure for (mis)matches

After all scopes are applied, it might turn out no matches were found. The default behavior is to silently succeed:

$ echo 'Some input...' | srgn --delete '\d'
Some input...

The output matches the specification: all digits are removed. There just happened to be none. No matter how many actions are applied, the input is returned unprocessed once this situation is detected. Hence, no unnecessary work is done.

One might prefer receiving explicit feedback (exit code other than zero) on failure:

echo 'Some input...' | srgn --delete --fail-none '\d'  # will fail

The inverse scenario is also supported: failing if anything matched. This is useful for checks (for example, in CI) against "undesirable" content. This works much like a custom, ad-hoc linter.

Take for example "old-style" Python code, where type hints are not yet surfaced to the syntax-level:

def square(a):
    """Squares a number.

    :param a: The number (type: int or float)
    """

    return a**2

This style can be checked against and "forbidden" using:

cat oldtyping.py | srgn --python 'doc-strings' --fail-any 'param.+type'  # will fail

Literal scope

This causes whatever was passed as the regex scope to be interpreted literally. Useful for scopes containing lots of special characters that otherwise would need to be escaped.

Rust library

Contributing

See guidelines.

Similar tools

An unordered list of similar tools you might be interested in.

Comparison with tr

srgn is inspired by tr, and in its simplest form behaves similarly, but not identically. In theory, tr is quite flexible. In practice, it is commonly used mainly across a couple specific tasks. Next to its two positional arguments ('arrays of characters'), one finds four flags:

  1. -c, -C, --complement: complement the first array
  2. -d, --delete: delete characters in the first first array
  3. -s, --squeeze-repeats: squeeze repeats of characters in the first array
  4. -t, --truncate-set1: truncate the first array to the length of the second

In srgn, these are implemented as follows:

  1. is not available directly as an option; instead, negation of regular expression classes can be used (e.g., [^a-z]), to much more potent, flexible and well-known effect
  2. available (via regex)
  3. available (via regex)
  4. not available: it's inapplicable to regular expressions, not commonly used and, if used, often misused

To show how uses of tr found in the wild can translate to srgn, consider the following section.

Use cases and equivalences

The following sections are the approximate categories much of tr usage falls into. They were found using GitHub's code search. The corresponding queries are given. Results are from the first page of results at the time. The code samples are links to their respective sources.

As the stdin isn't known (usually dynamic), some representative samples are used and the tool is exercised on those.

Identifier Safety

Making inputs safe for use as identifiers, for example as variable names.

Query

  1. tr -C '[:alnum:]_\n' '_'

    Translates to:

    $ echo 'some-variable? 🤔' | srgn '[^[:alnum:]_\n]' '_'
some_variable___

    Similar examples are:

  2. tr -c '[:alnum:]' _

    Translates to:

    $ echo 'some  variablê' | srgn '[^[:alnum:]]' '_'
some__variabl_

  3. tr -c -s '[:alnum:]' '-'

    Translates to:

    $ echo '🙂 hellö???' | srgn -s '[^[:alnum:]]' '-'
-hell-

Literal-to-literal translation

Translates a single, literal character to another, for example to clean newlines.

Query

  1. tr " " ";"

    Translates to:

    $ echo 'x86_64 arm64 i386' | srgn ' ' ';'
x86_64;arm64;i386

    Similar examples are:

  2. tr '.' "\n":

    Translates to:

    $ echo '3.12.1' | srgn --literal-string '.' '\n'  # Escape sequence works
3
12
1
$ echo '3.12.1' | srgn '\.' '\n'  # Escape regex otherwise
3
12
1

  3. tr '\n' ','

    Translates to:

    $ echo -ne 'Some\nMulti\nLine\nText' | srgn --literal-string '\n' ','
Some,Multi,Line,Text

    If escape sequences remain uninterpreted (echo -E, the default), the scope's escape sequence will need to be turned into a literal \ and n as well, as it is otherwise interpreted by the tool as a newline:

    $ echo -nE 'Some\nMulti\nLine\nText' | srgn --literal-string '\\n' ','
Some,Multi,Line,Text

    Similar examples are:

Removing a character class

Very useful to remove whole categories in one fell swoop.

Query

  1. tr -d '[:punct:]' which they describe as:

    Omit all punctuation characters

    translates to:

    $ echo 'Lots... of... punctuation, man.' | srgn -d '[[:punct:]]'
Lots of punctuation man

Lots of use cases also call for inverting, then removing a character class.

Query

  1. tr -cd a-z

    Translates to:

    $ echo 'i RLY love LOWERCASING everything!' | srgn -d '[^[:lower:]]'
iloveeverything

  2. tr -cd 'a-zA-Z0-9'

    Translates to:

    $ echo 'All0wed ??? 💥' | srgn -d '[^[:alnum:]]'
All0wed

  3. tr -cd '[[:digit:]]'

    Translates to:

    $ echo '{"id": 34987, "name": "Harold"}' | srgn -d '[^[:digit:]]'
34987

Remove literal character(s)

Identical to replacing them with the empty string.

Query

  1. tr -d "."

    Translates to:

    $ echo '1632485561.123456' | srgn -d '\.'  # Unix timestamp
1632485561123456

    Similar examples are:

  2. tr -d '\r\n'

    Translates to:

    $ echo -e 'DOS-Style\r\n\r\nLines' | srgn -d '\r\n'
DOS-StyleLines

    Similar examples are:

Squeeze whitespace

Remove repeated whitespace, as it often occurs when slicing and dicing text.

Query

  1. tr -s '[:space:]'

    Translates to:

    $ echo 'Lots   of  space !' | srgn -s '[[:space:]]'  # Single space stays
Lots of space !

    Similar examples are:

  2. tr -s ' ' '\n' (squeeze, then replace)

    Translates to:

    $ echo '1969-12-28    13:37:45Z' | srgn -s ' ' 'T'  # ISO8601
1969-12-28T13:37:45Z

  3. tr -s '[:blank:]' ':'

    Translates to:

    $ echo -e '/usr/local/sbin \t /usr/local/bin' | srgn -s '[[:blank:]]' ':'
/usr/local/sbin:/usr/local/bin

Changing character casing

A straightforward use case. Upper- and lowercase are often used.

Query

  1. tr A-Z a-z (lowercasing)

    Translates to:

    $ echo 'WHY ARE WE YELLING?' | srgn --lower
why are we yelling?

    Notice the default scope. It can be refined to lowercase only long words, for example:

    $ echo 'WHY ARE WE YELLING?' | srgn --lower '\b\w{,3}\b'
why are we YELLING?

    Similar examples are:

  2. tr '[a-z]' '[A-Z]' (uppercasing)

    Translates to:

    $ echo 'why are we not yelling?' | srgn --upper
WHY ARE WE NOT YELLING?

    Similar examples are:

[^1]: Currently, reversibility is not possible for any other action. For example, lowercasing is not the inverse of uppercasing. Information is lost, so it cannot be undone. Structure (imagine mixed case) was lost. Something something entropy...