breakmancer

Drop a breakpoint into any shell.

Need to debug a build script, but don't have a way to SSH into the server to poke around in the build environment—or you do, but you need a way to pause the script at exactly the right point? breakmancer allows you to do just that!

Functionally, this is a reverse shell for people who are authorized to jump into a remote server. Unlike the malicious kind of reverse shell, this one aims to preserve the security of the target.

Requirements

• A development computer that is capable of listening on a public IP address (or an intranet IP if it's on the same network as the computer you're debugging.)
• A target computer where you're trying to debug a script and that you have permission to shell into. This computer must be able to reach out to the development computer over TCP.

Usage

In this example, the development computer is reachable on port 12345 at the domain name home.timmc.org. The target is a GitHub Actions workflow.

1. Start controller session locally: On my laptop I run breakmancer start home.timmc.org:12345. It prints out instructions and a command line and waits for a connection.
2. Set up the breakpoint: I add the breakpoint command that was printed out above to my GitHub workflow file. It might look like this:

   name: Breakmancer Demo
   on: [push]
   jobs:
     demo:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
   
         - name: Do some stuff
           run: |
             date > something.txt
   
         - name: Invoke breakpoint
           run: |
             # If I want to use the hosted version:
             curl -sS https://codeberg.org/timmc/breakmancer/raw/branch/master/fetch-and-run.sh \
               | bash -s -- break --auth dual-pub -c home.timmc.org:12345 -i 4TXapNw9DLbnjhFx7
   
             # Or if I want to use a copy I've temporarily checked into git:
             chmod +x ./breakmancer
             ./breakmancer break --auth dual-pub -c home.timmc.org:12345 -i 4TXapNw9DLbnjhFx7
   

3. Run: I push the branch, the GitHub Action runs, and breakmancer reaches back to my laptop.
4. Verify: The breakpoint prints out a verification string like xWeBv1Qw2BnxEBuQ and instructs me to paste it into the controller when prompted.
5. Use the shell: A shell opens, and I can run commands on the breakpoint from the controller session:

   Connection from [::ffff:52.234.38.67]:61440
   Session ready. You can enter single-line commands. Use `exit` to exit.
   >> hostname
   [out] fv-az1756-422
   [exit: 0]
   >> pwd
   [out] /home/runner/work/manual-testing/manual-testing
   [exit: 0]
   >>
   

6. End: When I'm done, I use exit (or ^C or ^D) to exit and allow the workflow to continue; at the new prompt I can choose to wait for a new connection or end the program entirely.
7. Audit log: Looking at the GitHub workflow output, I see a log of what commands were run:

   [2025-04-28T00:23:52Z] Breakpoint will attempt to connect to controller at 'localhost:12345', expecting controller identity of '4TXapNw9DLbnjhFx7'.
   Connecting to controller at 12.34.56.78:12345
   [2025-04-28T00:23:52Z] Waiting for first command...
   [2025-04-28T00:23:57Z] Running command: hostname
   [2025-04-28T00:23:59Z] Running command: pwd
   Controller asked for normal execution to resume; exiting breakpoint.
   

Tip: Once the controller has exited, the secret and command line given to the breakpoint side will no longer work and will need to be replaced. That can be annoying. So until you're sure that you're done, you may want to leave the controller running.

Build, test, and release

• Review changelog to ensure nothing important has been omitted
• Update version in Cargo.toml
• Update the changelog version (add a new release header below the Unreleased header)
• Update the version in fetch-and-run.sh
• cargo clippy --all-targets && cargo test && cargo doc --no-deps
• cargo build --release
• Do any manual testing
• Commit version change
• cargo publish
• git tag -a vX.Y.Z -m X.Y.Z
• git push && git push --tags
• Cut a new release on Codeberg:
  • Name the release X.Y.Z (no v prefix)
  • Attach ./target/release/breakmancer to the release
  • Hide auto-generated archives
• Test the release by running ./fetch-and-run.sh

Manual testing

Some interesting things to enter:

• (yes "out" | head -n20)& (yes " err" | head -n20 >&2) to show interleaving of stdout and stderr
• echo start; sleep 10; echo stop to experiment with blocking commands
• while true; do date; sleep 0.001; done for a rapid output stream (not as fast as yes)

Limitations

• As breakmancer does not create a PTY, buffering can occur in some pipelines. This creates two problems:

  1. All of the output happens at once. In the command for x in {1..5}; do date; sleep 1; done | head we see nothing for 5 seconds, and then 5 lines of output all at once. (The output reveals that each date invocation did indeed occur 1 second apart.)
  2. If such a streaming command never terminates, the output is never sent. The command while true; do date; sleep 1; done | head will hang forever with no output.

  In those examples, dropping head from the pipeline (or piping to a different binary such as cat) allows the output to stream normally.

• See TODO.md for more bugs and wanted features.

License

Copyright 2024-2025 Tim McCormack

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.