# OpenCC Source Layout
This directory contains the C++ core library, the public C API, command-line
tools, and native extension entry points. The code is organized around a simple
conversion pipeline:
1. Load a JSON configuration.
2. Load segmenters and dictionaries referenced by that configuration.
3. Segment input text.
4. Apply one or more dictionary-based conversion stages.
5. Return converted text or inspection data.
The data files that drive this pipeline live outside this directory:
- `data/config/*.json`: conversion schemes.
- `data/dictionary/*.txt`: source dictionaries.
- generated `.ocd2` files: marisa-trie dictionary binaries.
## Main Pipeline
### Configuration
- `Config.hpp`, `Config.cpp`
- Finds and parses JSON configuration files.
- Builds segmenters, dictionaries, dictionary groups, conversions, and
converters.
- Resolves dictionary/resource files through `ResourceProvider`.
- Keeps config loading separate from dictionary resource lookup.
- Uses UTF-16-capable file checks on Windows for internal config and
resource paths.
- `ResourceProvider.hpp`, `ResourceProvider.cpp`
- Provides `ResourceProvider` and `FilesystemResourceProvider`.
- `FilesystemResourceProvider` searches configured resource directories in
order and returns a resolved path for lower-level dictionary loaders.
- Embedded hosts can combine application, plugin, and system OpenCC data
directories without changing config JSON.
### Segmentation
- `Segmentation.hpp`, `Segmentation.cpp`
- Base interface for segmenters.
- `MaxMatchSegmentation.hpp`, `MaxMatchSegmentation.cpp`
- Maximum forward matching segmenter used by normal OpenCC configs.
- Uses dictionary prefix matches to keep phrases intact.
- `PluginSegmentation.hpp`, `PluginSegmentation.cpp`
- Runtime-loaded segmentation plugin support.
- Used by the Jieba plugin.
- `Segments.hpp`
- Segment container used between segmenters, conversions, and inspection
output.
### Conversion
- `Converter.hpp`, `Converter.cpp`
- Owns a segmenter and a conversion chain.
- Provides high-level string and buffer conversion.
- `ConversionChain.hpp`, `ConversionChain.cpp`
- Ordered list of conversion stages.
- `Conversion.hpp`, `Conversion.cpp`
- One dictionary-backed conversion stage.
- `ConversionInspection.hpp`
- Data structures returned by inspection mode.
The core conversion path depends on segmentation and longest-prefix dictionary
matching. Character-by-character replacement is not equivalent to OpenCC
behavior because phrase priority and multi-stage conversion order matter.
## Dictionaries
### Interfaces and Shared Types
- `Dict.hpp`, `Dict.cpp`
- Abstract dictionary interface.
- Supports exact match, prefix match, all-prefix match, and enumeration.
- `DictEntry.hpp`, `DictEntry.cpp`
- Key/value entry representation.
- `PrefixMatch.hpp`, `PrefixMatch.cpp`
- Prefix match result.
- `Lexicon.hpp`, `Lexicon.cpp`
- In-memory collection of dictionary entries.
- `SerializableDict.hpp`
- Template helpers for loading serialized dictionaries from files.
- `SerializedValues.hpp`, `SerializedValues.cpp`
- Compact storage for candidate value lists used by `.ocd2`.
### Implementations
- `TextDict.hpp`, `TextDict.cpp`
- Tab-delimited text dictionary.
- Useful for source data and tests.
- `MarisaDict.hpp`, `MarisaDict.cpp`
- Default `.ocd2` dictionary format.
- Uses marisa-trie for compact prefix lookup.
- `DartsDict.hpp`, `DartsDict.cpp`
- Legacy `.ocd` dictionary format.
- Requires Darts support.
- `BinaryDict.hpp`, `BinaryDict.cpp`
- Legacy binary payload support used with Darts serialization.
- `DictGroup.hpp`, `DictGroup.cpp`
- Ordered group of dictionaries.
- Tries dictionaries in sequence and returns the first usable match.
- `DictConverter.hpp`, `DictConverter.cpp`
- Converts dictionary files between supported formats.
### Binary Format Portability
Neither binary dictionary format is portable across platforms, but they
fail differently, and the constraints differ per format.
| `.ocd` | dependent | **independent** (since 1.4.1) |
| `.ocd2` | dependent | dependent |
- **`.ocd` is word-size independent.** The Darts double array uses
`id_type = unsigned int` (4 bytes on every supported platform), and
`DartsDict::SerializeToFile` writes the array length as an explicit
`uint32_t`. `BinaryDict` writes all of its integer fields as fixed-width
`uint32_t` as well. Files written by pre-1.4.1 64-bit builds used native
8-byte `size_t` fields; `BinaryDict::DetectFieldWidth()` still recognizes
that legacy layout, and `LooksLikeLegacy64Field()` performs the probe
through a native `uint64_t` read so the detection is correct on
big-endian hosts too.
- **`.ocd2` is not.** The `SerializedValues` payload OpenCC writes uses
fixed-width `uint32_t` fields, but the marisa-trie image that precedes it
does not: `MARISA_WORD_SIZE` is derived from `UINTPTR_MAX`, and the
serialized `BitVector::Unit` / `FlatVector::Unit` are `uint64_t` on
64-bit builds and `uint32_t` on 32-bit builds. A 64-bit-built `.ocd2`
loaded by a 32-bit build is not reliably rejected either — marisa's
`Vector<T>::map` only checks `total_size % sizeof(T)`, and a multiple of
8 is always a multiple of 4 — so the mismatch can be read as silently
wrong data rather than an error.
Forcing `MARISA_WORD_SIZE` to 32 would not make `.ocd2` portable, since
byte order would still differ; it would not apply to `USE_SYSTEM_MARISA`
builds (which distributions use), where the format would then depend on
which marisa the build linked against; and it would narrow the rank/select
word width in `MarisaDict`'s lookup hot path. Making `.ocd2` genuinely
portable means normalizing the trie image on write and converting back on
load, which gives up the in-place mmap that makes the format fast.
The practical consequence is that generated `.ocd2` files must be built
for the architecture that consumes them, and must never be shipped from an
architecture-independent package — see `packaging/debian/README.md` for
how this played out in Debian.
## Public APIs
### C++ API
- `SimpleConverter.hpp`, `SimpleConverter.cpp`
- High-level C++ wrapper around `Config` and `Converter`.
- Accepts a config name/path, optional search paths, or an explicit
`ResourceProvider`.
- Throws C++ exceptions on failures.
### C API
- `opencc.h`
- Stable C ABI for downstream users and bindings.
- Exposes `opencc_open`, `opencc_open_w`, `opencc_convert_utf8`,
`opencc_convert_utf8_free`, `opencc_close`, and error helpers.
Windows path semantics need care:
- `opencc_open_w(const wchar_t*)` is the explicit UTF-16 Windows API.
- `opencc_open(const char*)` keeps the historical Windows/MSVC narrow-string
behavior. Do not silently change its encoding contract without a migration
plan.
- New Windows path-taking C APIs should use explicit names such as
`*_utf8` or `*_w` rather than relying on ambiguous `char*` semantics.
### Python Extension
- `py_opencc.cpp`
- Python module binding built on top of the C++ core.
## Command-Line Tools
The native CLI tools live under `src/tools`.
- `CommandLine.cpp`
- Small executable entry point.
- On Windows/MSVC, obtains command-line arguments through wide Windows APIs
and converts them to UTF-8 before dispatching to the CLI core.
- `CommandLineMain.hpp`, `CommandLineMain.cpp`
- Main `opencc` command implementation.
- Handles conversion, segmentation/inspection modes, measurement output,
stream conversion, and explicit `--in-place` conversion.
- `PlatformIO.hpp`, `PlatformIO.cpp`
- CLI platform boundary for path-sensitive operations.
- Handles UTF-8 file open/remove/replace, temporary output files, same-file
detection, real path resolution, and Windows argv collection.
- `DictConverter.cpp`
- `opencc_dict` implementation.
- `PhraseExtract.cpp`
- `opencc_phrase_extract` implementation.
- `CmdLineOutput.hpp`
- Shared command-line help formatting.
CLI file conversion must remain streaming. Do not replace stream processing
with "read whole file into memory" logic. In-place conversion is intentionally
opt-in:
- Without `--in-place`, `-i` and `-o` referring to the same actual file is
rejected.
- With `--in-place`, output is written to a temporary file next to the target,
then the target is replaced after conversion succeeds.
## Platform and Path Handling
OpenCC uses UTF-8 internally for paths unless an API explicitly documents a
different contract. Windows code should convert to UTF-16 at the platform
boundary.
Important files:
- `UTF8Util.hpp`, `UTF8Util.cpp`
- UTF-8 helpers and platform string conversion helpers.
- `WinUtil.hpp`
- Small Windows-only UTF-8/UTF-16 conversion helpers.
- `tools/PlatformIO.*`
- CLI path boundary.
Maintenance rules:
- Do not add raw `fopen`, `std::fstream`, `stat`, `std::tmpnam`, or narrow
Win32 path calls to new Windows-sensitive paths.
- Prefer existing platform helpers or add a clearly named boundary helper.
- Keep public API encoding contracts explicit.
- Validate Windows path changes with real Windows CI. Zig cross-compilation is
useful for compile/link coverage, but it does not execute Windows runtime
path behavior.
## Plugins
- `plugin/OpenCCPlugin.h`
- C plugin ABI used by loadable segmentation plugins.
- `PluginSegmentation.*`
- Host-side plugin loader and adapter.
The Jieba plugin lives outside `src` under `plugins/jieba`.
## Tests
Most core modules have adjacent `*Test.cpp` files in this directory. Important
test groups include:
- `ConfigTest.cpp`
- Config search paths and Unicode config path handling.
- `SimpleConverterTest.cpp`
- C++ wrapper behavior and C API basics.
- `Conversion*Test.cpp`
- Conversion and inspection behavior.
- `MaxMatchSegmentationTest.cpp`
- Segmenter behavior.
- `MarisaDictTest.cpp`, `TextDictTest.cpp`, `DictGroupTest.cpp`
- Dictionary implementations.
- `UTF8*Test.cpp`
- UTF-8 slicing and utility behavior.
CLI tests live in `test/CommandLineConvertTest.cpp` because they execute the
built command-line binary. They cover streaming behavior, Unicode paths,
measurement output, inspection modes, and in-place conversion safety.
## Build Integration
The source tree is built through both CMake and Bazel:
- `src/CMakeLists.txt`
- `src/BUILD.bazel`
- `src/tools/CMakeLists.txt`
- `src/tools/BUILD.bazel`
When adding or renaming source files, update both build systems and any direct
cross-build scripts that carry explicit source lists.