Expand description
Apache Spark (spark-submit) log compression.
Spark drivers emit hundreds of YY/MM/DD HH:MM:SS INFO Component: ...
lines. We drop INFO noise, keep finished-job lines, deduplicate WARNs and
preserve ERRORs / exceptions, then prefix a one-line job/warn/error count.
Crucially we also keep lines that are NOT framework log records — these are
the application’s own stdout (e.g. Result: total words = 184273), which is
the actual point of the run and must never be dropped.