Tokenize

Technically nothing is "tokenized", all files are simply put into one file along with some formatting and an initial prompt.

Usage

Basic Usage

tokenize [TARGET_DIR]

Specify a custom output file

tokenize . -o output.md

[INOP] Specify a custom prompt for the beginning of the file

tokenize . -p prompt.txt

Dont respect .gitignore files

tokenize [TARGET_DIR] --no-gitignore

Include hidden files

tokenize [TARGET_DIR] --include-hidden

Follow Symlinks This was not tested, I am not sure if it actually includes the files in the final output

tokenize [TARGET_DIR] --follow symlinks

Attribution

• ignore