Using strace with SDP-Managed Services

Overview

The SDP init scripts p4d_base, p4broker_base, and p4p_base include built-in support for running a service under strace at startup. This provides a repeatable, low-friction way to capture syscall traces for diagnostic purposes without modifying any init script or configuration file.

Note: strace -f on a busy server generates large output volumes and adds measurable overhead. This feature is intended for short-term diagnostic use only — not for permanent production deployment.

Two strace Modes

The init scripts automatically select one of two modes based on whether the service is managed by systemd or started directly (non-systemd).

Wrap Mode (systemd — complete capture)

When a service is managed by systemd, strace is interposed between the init script and the service binary:

systemd → p4d_N_init → strace ... p4d $P4D_FLAGS

strace runs the service binary as its own child process. Every syscall is captured from the very first instruction — nothing from startup is missed. This is ideal for diagnosing issues that occur during early startup (e.g. replication thread initialization on a replica).

Trade-off: Because strace is the parent process, the service cannot be stopped independently from strace. Stop the service normally (systemctl stop p4d_N) to terminate both.

Attach Mode (non-systemd — best effort)

When a service is started directly (without systemd), the init script starts the service normally and then attaches strace using strace -p <pid> once the service process ID is known.

Trade-off: Syscalls that occur between process start and the moment the PID is discovered (typically a second or less) are not captured. For services where startup-phase syscalls are important, use systemd wrap mode.

strace runs as a completely independent background process. It can be stopped at any time without affecting the running service.

File Naming Convention

All strace-related files live in $LOGS and incorporate the service name (e.g. p4d_1, p4broker_1, p4p_1) so that multiple instances on the same host never collide.

The strace output log also includes a timestamp (generated at the moment strace starts) so successive runs do not overwrite each other.

| File | Purpose | |------|---------| | $LOGS/.strace.<ServiceName> | Marker file — create this to enable strace | | $LOGS/.strace.<ServiceName>.pid | PID of the running strace process (attach mode only) | | $LOGS/strace.<ServiceName>.<YYYYMMDD_HHMMSS>.log | strace output |

For p4d instance 1 on a host where $LOGS is /p4/1/logs:

/p4/1/logs/.strace.p4d_1 ← marker file (you create this) /p4/1/logs/.strace.p4d_1.pid ← strace PID (attach mode only) /p4/1/logs/strace.p4d_1.20260627_143022.log ← strace output

For a p4broker on the same instance:

/p4/1/logs/.strace.p4broker_1 /p4/1/logs/.strace.p4broker_1.pid /p4/1/logs/strace.p4broker_1.20260627_143055.log

A p4broker started with a non-default config tag (e.g. mytag) uses p4broker_1_mytag as the service name.

Enabling strace — Step by Step

p4d (Helix Core Server)

With systemd (wrap mode — recommended for startup diagnosis):

```bash

1. Create the marker file (empty = use default flags: -f -tt)

touch /p4/N/logs/.strace.p4d_N

2. (Re)start the service

sudo systemctl restart p4d_N

3. Confirm strace is running (the service log will show the strace log path)

grep strace /p4/N/logs/p4d_init.log | tail -5 ```

Without systemd (attach mode):

```bash touch /p4/N/logs/.strace.p4dN /p4/N/bin/p4dN_init start

Confirm strace is running

cat /p4/N/logs/.strace.p4dN.pid ps -p $(cat /p4/N/logs/.strace.p4dN.pid) ```

p4broker (Helix Broker)

```bash touch /p4/N/logs/.strace.p4broker_N

sudo systemctl restart p4broker_N # systemd (wrap mode)

or

/p4/N/bin/p4brokerNinit start # non-systemd (attach mode) ```

p4p (Helix Proxy)

```bash touch /p4/N/logs/.strace.p4p_N

sudo systemctl restart p4p_N # systemd (wrap mode)

or

/p4/N/bin/p4pNinit start # non-systemd (attach mode) ```

Stopping strace Without Affecting the Service

Attach mode only. Because strace runs as a separate background process in attach mode, it can be killed independently at any time. The service continues running normally.

```bash

Generic form:

kill $(cat /p4/N/logs/.strace..pid)

Examples:

kill $(cat /p4/1/logs/.strace.p4d1.pid) kill $(cat /p4/1/logs/.strace.p4broker1.pid) kill $(cat /p4/1/logs/.strace.p4p_1.pid) ```

The init script also calls stop_strace_if_running automatically when the service is stopped via the init script, cleaning up the PID file.

Note: In attach mode, strace also exits automatically when its target process dies, so no explicit cleanup is needed if the service itself stops.

Wrap mode: strace is the parent process. The only way to stop it is to stop the service normally (e.g. systemctl stop p4d_N). Killing strace directly will also stop the service.

Preventing strace from Running on Next Restart

Remove the marker file before restarting the service:

bash rm /p4/N/logs/.strace.p4d_N sudo systemctl restart p4d_N

Custom strace Flags

By default, strace is invoked with -f -tt (follow forks, include timestamps with microsecond resolution). To use different flags, write them to the marker file on a single line before starting the service:

```bash

Trace only network-related syscalls:

echo '-f -e trace=network' > /p4/1/logs/.strace.p4d_1

Trace file-related syscalls with string lengths up to 256 bytes:

echo '-f -s 256 -e trace=file' > /p4/1/logs/.strace.p4d_1

Default (empty file):

touch /p4/1/logs/.strace.p4d_1 ```

The flags in the marker file replace the defaults entirely, so include -f explicitly if you want fork-following.

Viewing the Output

The exact log file path (including the timestamp) is recorded in the service init log (p4d_init.log, p4broker_init.log, or p4p_init.log) at the moment strace starts.

```bash

Find the strace log path for the most recent run:

grep 'strace' /p4/1/logs/p4d_init.log | grep 'output:'

Follow the trace in real time (strace writes line-buffered with -tt):

tail -f /p4/1/logs/strace.p4d1.20260627143022.log

Count syscalls by name:

sort /p4/1/logs/strace.p4d1.20260627143022.log | uniq -c | sort -rn | head -20 ```

Implementation Notes

Shared Library

The shared library /p4/common/lib/strace.lib is sourced by all three init scripts. It provides three functions:

| Function | Purpose | |----------|---------| | run_service_under_strace <binary> [args...] | Wrap mode: run service as strace child | | attach_strace_if_requested <pid_cmd> | Attach mode: attach strace to running service | | stop_strace_if_running | Stop background strace and remove PID file |

Mode Selection

Mode is selected automatically based on $UseSystemd:

$UseSystemd == 1 → wrap mode (run_service_under_strace) $UseSystemd == 0 → attach mode (attach_strace_if_requested ... & disown)

UseSystemd is set to 1 when a systemd unit file for the service exists and is active (detected at script startup via systemctl).

ServiceName

The ServiceName variable (e.g. p4d_1) is derived from the service binary name (e.g. ${P4DBIN##*/}) and is set unconditionally before any systemd detection, so strace file naming works correctly on both systemd and non-systemd hosts.

Why Not Always Use Wrap Mode?

In non-systemd operation, the service binary forks immediately and daemonizes. Using strace binary args (wrap mode) in this case would cause strace to block indefinitely tracking the daemon, holding the init script open. The attach approach is better suited here: the init script exits normally, and strace attaches in a separate background process.

With systemd, the init script is already meant to block (the service runs in the foreground as ExecStart). Replacing binary args with strace ... binary args is a natural fit — no additional complexity is introduced, and systemd manages the strace process as the service.