Change 33340
Add structured logging (JSONL/Prometheus) to SDP maintenance scripts
Failure has historically gone silently unlogged in these scripts while
success was well-logged: verify_shelves.sh had no error detection at
all, most other scripts only log a human-readable narrative with
nothing machine-readable, and nothing here is consumable by
monitoring tools (telegraf, Datadog, Prometheus) without a bespoke
parser per script.
Adds one shared bash helper (sdp_structured_log.sh, sourced
automatically by backup_functions.sh) and one shared stdlib-only
Python module (sdp_structured_log.py, importable by both cron scripts
and trigger scripts). One mode at a time, selected by the new
SDP_LOG_FORMAT p4_vars setting (jsonl by default, or prom for sites
running Prometheus instead, or off) - never simultaneous dual-writing.
jsonl mode appends one JSON object per run alongside each script's
existing .log file; prom mode writes Prometheus textfile-collector
gauges to $SDP_METRICS_DIR (default /p4/metrics, matching the existing
p4prometheus_metrics_dir convention in the perforce-sdp-monitoring
role) using the same atomic write-then-mv convention that role's own
health-check scripts already use.
Success/failure capture is trap-based (bash: EXIT/ERR/HUP/INT/TERM,
consolidated so exactly one terminal record is emitted regardless of
how a script exits) / atexit-based (Python), specifically so a script
dying unexpectedly - an unbound variable, a set -e failure, a signal -
still gets recorded rather than silently vanishing. die() is annotated
(2 lines) rather than made to emit directly, so every one of the ~40
scripts that source backup_functions.sh gets failure-path coverage
automatically once they call the new sdp_log_init, with no per-script
failure plumbing needed.
Also fixes verify_shelves.sh, which had no error detection whatsoever:
`p4 changes`'s exit code was discarded and consumed by an unquoted
`for` loop, so a failed listing produced an empty loop and exit 0 - a
total failure was indistinguishable from "no shelves to verify". `p4
verify`'s own exit code was likewise never checked, and the script
rm -f'd its own logfile every run (after log_init had already started
tailing it interactively, which broke that too). Added the missing
check_vars/set_vars/check_uid/p4login preflight that every other
script here already has.
Restructures the run_if_{master,edge,replica,broker,proxy}.sh cron
wrapper family: each previously ended with `exec "$@"`, which replaces
the wrapper's own process image, so no trap could ever fire and this
wrapper's own success/failure was structurally impossible to record.
Now a normal call + $? capture, still propagating the wrapped command's
exit code unchanged. This alone gives structured coverage for every
cron job routed through these wrappers, before any individual wrapped
script is itself instrumented. Also fixes a related silent gap in the
master/edge/replica variants: SERVER_TYPE matching neither the proxy
skip nor the expected role fell through to an unexplained implicit
exit 1 - now recorded as an explicit failure naming the mismatch,
instead of a bare unexplained nonzero exit cron would mail with no
context.
triggers/keep_group_unset.py: fixed its stale `#!/usr/bin/env python`
shebang to python3 (the body already requires it via os.replace(),
3.3+ only), and added a guarded structured-logging record. Behavior
preservation is deliberate here: an unexpected exception still rejects
the p4 group form edit exactly as before (fail-closed) - the change
only adds a logged record explaining why, never stdout (p4d relays a
trigger's stdout to the connected client), and degrades to a silent
no-op if sdp_structured_log.py hasn't been deployed yet (it ships in
the same tarball as this trigger but via a different deploy path than
the Ansible-templated copy of this same file - see the update-tgz.sh
changelist that follows this one).
Extends rotate_last_run_logs/remove_old_logs to cover the new .jsonl
siblings, including adding verify_shelves.log/.jsonl to the KEEPLOGS
cleanup list - it was absent from that list entirely before (masked
by the rm -f bug this same change removes), so without this addition
it would have grown unbounded.
Failure has historically gone silently unlogged in these scripts while
success was well-logged: verify_shelves.sh had no error detection at
all, most other scripts only log a human-readable narrative with
nothing machine-readable, and nothing here is consumable by
monitoring tools (telegraf, Datadog, Prometheus) without a bespoke
parser per script.
Adds one shared bash helper (sdp_structured_log.sh, sourced
automatically by backup_functions.sh) and one shared stdlib-only
Python module (sdp_structured_log.py, importable by both cron scripts
and trigger scripts). One mode at a time, selected by the new
SDP_LOG_FORMAT p4_vars setting (jsonl by default, or prom for sites
running Prometheus instead, or off) - never simultaneous dual-writing.
jsonl mode appends one JSON object per run alongside each script's
existing .log file; prom mode writes Prometheus textfile-collector
gauges to $SDP_METRICS_DIR (default /p4/metrics, matching the existing
p4prometheus_metrics_dir convention in the perforce-sdp-monitoring
role) using the same atomic write-then-mv convention that role's own
health-check scripts already use.
Success/failure capture is trap-based (bash: EXIT/ERR/HUP/INT/TERM,
consolidated so exactly one terminal record is emitted regardless of
how a script exits) / atexit-based (Python), specifically so a script
dying unexpectedly - an unbound variable, a set -e failure, a signal -
still gets recorded rather than silently vanishing. die() is annotated
(2 lines) rather than made to emit directly, so every one of the ~40
scripts that source backup_functions.sh gets failure-path coverage
automatically once they call the new sdp_log_init, with no per-script
failure plumbing needed.
Also fixes verify_shelves.sh, which had no error detection whatsoever:
`p4 changes`'s exit code was discarded and consumed by an unquoted
`for` loop, so a failed listing produced an empty loop and exit 0 - a
total failure was indistinguishable from "no shelves to verify". `p4
verify`'s own exit code was likewise never checked, and the script
rm -f'd its own logfile every run (after log_init had already started
tailing it interactively, which broke that too). Added the missing
check_vars/set_vars/check_uid/p4login preflight that every other
script here already has.
Restructures the run_if_{master,edge,replica,broker,proxy}.sh cron
wrapper family: each previously ended with `exec "$@"`, which replaces
the wrapper's own process image, so no trap could ever fire and this
wrapper's own success/failure was structurally impossible to record.
Now a normal call + $? capture, still propagating the wrapped command's
exit code unchanged. This alone gives structured coverage for every
cron job routed through these wrappers, before any individual wrapped
script is itself instrumented. Also fixes a related silent gap in the
master/edge/replica variants: SERVER_TYPE matching neither the proxy
skip nor the expected role fell through to an unexplained implicit
exit 1 - now recorded as an explicit failure naming the mismatch,
instead of a bare unexplained nonzero exit cron would mail with no
context.
triggers/keep_group_unset.py: fixed its stale `#!/usr/bin/env python`
shebang to python3 (the body already requires it via os.replace(),
3.3+ only), and added a guarded structured-logging record. Behavior
preservation is deliberate here: an unexpected exception still rejects
the p4 group form edit exactly as before (fail-closed) - the change
only adds a logged record explaining why, never stdout (p4d relays a
trigger's stdout to the connected client), and degrades to a silent
no-op if sdp_structured_log.py hasn't been deployed yet (it ships in
the same tarball as this trigger but via a different deploy path than
the Ansible-templated copy of this same file - see the update-tgz.sh
changelist that follows this one).
Extends rotate_last_run_logs/remove_old_logs to cover the new .jsonl
siblings, including adding verify_shelves.log/.jsonl to the KEEPLOGS
cleanup list - it was absent from that list entirely before (masked
by the rm -f bug this same change removes), so without this addition
it would have grown unbounded.
Tip: Use n and p to cycle through the changes.