fstool tech-support is the Perl script /usr/local/forescout/lib/fstool/commands/tech-support.pl. Most of its helpers live in lib/perl/forescout/support.ph (the sup_* functions, including the network send). It does three jobs:
summary.txt and errors/..tgz, or send to the Forescout snapshot server.The first argument picks a subcommand. No argument, or an argument starting with -, runs default (collect a snapshot). The others:
| Subcommand | What it does |
|---|---|
| (default) | Collect, analyze, then send, pack or review |
| send FILE... | Send arbitrary files (full paths) to Forescout support, with a system info summary |
| diag TGZ|DIR | Re-run the analysis on a snapshot someone else took, offline |
| version | Platform version plus the installed plugin table with plugin IDs |
| debug | Trace configuration utility (debug -l lists current trace settings) |
| stats ... | Statistics analysis utility |
| strace PID | strace a process into a log directory |
| jstack | Java stack sampling and histogram |
| realmem / meminfo | Real memory use per process |
| policy | Policy dump used inside the snapshot |
| deployment_summary | Deployment summary text |
| time DATE | Show how a date string parses to epoch (test your -t values) |
| perlstack PID | Perl stack of a running Perl process |
With no mode flag the tool asks interactively:
| Flag | Result |
|---|---|
| --pack | Collect, then tar and gzip to a file. Non-interactive. Use this for scripts and for your own server. |
| --send | Collect, then upload to the Forescout snapshot server. If the send fails it offers to pack instead. Over 2,000 MB it asks before sending. |
| --review | Collect and analyze, print the error table, keep the directory, no pack and no send. Not available from the restricted CLI. |
| --health-check | Same as --group health-check --review --show-errors --disable-sysbase. Fast yes or no on crashes, stalls, reboots, OOM, cores. |
| --company NAME | Company label; stored once in property fs.snapshot.name and reused next time. |
| --comment "TEXT" | Comment written into the summary and the send description. |
Default is the last 24 hours. One -t sets the start; two set start and end. Values are a period (7d, 6h, 30m), a date string, or utc:EPOCH.
fstool tech-support -t 7d # last 7 days fstool tech-support -t 7d -t 5d # from 7 days ago to 5 days ago fstool tech-support -t 9:00 # since 09:00 today fstool tech-support -t "Oct 20 12:00" -t "Oct 20 17:00" # one afternoon fstool tech-support -t utc:<EPOCH> # since an exact epoch fstool tech-support time "Oct 20 12:00" # check how a date parses
-t utc:<CRASH_EPOCH-3600> -t utc:<CRASH_EPOCH+600>. A tight window keeps the file small and makes the crash analysis focus on the event.A snapshot is a list of categories. Each has a callback that collects or analyzes, plus an XML file at plugin/support/items/<name>.xml that lists extra files and tables to attach.
| Category | Group | Default | What it does |
|---|---|---|---|
| service_diag | health-check | yes | CounterACT service crash analysis (exit status, OOM, killed, clock moved back, too many open files, stalls) |
| servicedelay | health-check | yes | Age of log/status; error if over fs.service.delay.thresh (30 s) |
| system_reboot | health-check | yes | Reboot events in the window |
| plugin | health-check | yes | Plugin crash detection (logic in section 8) |
| plugin_config_change | health-check | yes | Plugin restarts caused by configuration changes |
| local_patches | health-check | yes | Locally patched plugin files |
| plugin_stalled | health-check | yes | Plugins with queue delay; can attach strace samples |
| core | health-check | yes | Core files found (attach with --cores) |
| linux_oom | health-check | yes | Kernel OOM-killer events |
| ha_failover | health-check | yes, HA only | HA failover events |
| service | logs | yes | Service logs, fstool uptime, policy dump |
| plugin_curr_log | logs | yes | Current log of every plugin |
| data_sharing_telemetry | logs | yes | Telemetry JSON logs if data sharing is on |
| packet_engine | logs | yes, appliances | Packet engine logs and pf_ring filters |
| portal, events, stats | logs | yes | Portal logs, events, stats files |
| sys | data | yes | System commands (list in section 7), all etc/*.xml and *.properties |
| config | config | yes | All etc/*.xml and *.properties |
| filerep | config | yes | Repository files |
| jstack | data | yes | Java stack traces |
| db | data | yes | Database info and attached tables |
| stats_diag | data | review only | Statistics analysis; also runs in diag |
| hostinfo_diag | data | no | hostinfo table diagnostics (slow) |
| file_attrs | data | no | Attributes of all files |
| deployment_summary | data | no | Deployment summary (slow); shortcut --deployment-summary |
fstool tech-support --list-items # categories on this box fstool tech-support --list-groups # groups fstool tech-support --custom --include service_diag --include plugin # only these fstool tech-support --group health-check # one group fstool tech-support --exclude jstack --exclude db # defaults minus some fstool tech-support --include-files '<GLOB>' --exclude-files '<GLOB>' # file filters
-p <plugin_id> adds a per-plugin item that collects that plugin's logs, config and tables from its own support.xml. With --custom, -p also limits crash analysis to those plugins. --all-plugins adds every installed plugin.
fstool tech-support version # plugin IDs: sw, dot1x, va, ad, ... fstool tech-support -p sw -t 2d --pack fstool tech-support -p sw -p dot1x --pack
| Flag | Adds |
|---|---|
| --cores | Core files (large) |
| --dbtable TABLE | A database table dump; repeat for more. --dblisttables lists them. |
| --hinfo IP | IP-IP | fstool hostinfo and the raw hostinfo rows for one endpoint or a range |
| --attach-file /PATH | Any file or directory, full path; repeat for more |
| --properties FILE | Load tech-support settings from a properties file |
| --define key=value | Internal tuning, e.g. plugin_debug_level=5, capture_enabled=0, plugin_crash_last_lines=3000 |
| --anonymize | Anonymized .zip instead of .tgz. Cannot be combined with --send. |
| --show-errors | Print each category status and the error table on screen |
| Item | Default and control |
|---|---|
| Working directory | /tmp/snapshot.<PID>.<EPOCH>. -o /full/path picks another (must not exist). --tmp /full/path moves the temp root. |
| Packed file | <TMP>/<COMPANY>-v<VERSION>-<appliance|em>-<TIMESTAMP>.tgz. --packname /full/path/name.tgz sets it exactly (must end .tgz or .tar.gz). |
| Restricted CLI | From fstool cli the pack goes to /usr/local/forescout/log/shared/ and shows as shared:<name>. |
| Disk guard | Packing stops and deletes the file if it would use more than 90 percent of the free space it started with. Fix with --tmp on a bigger filesystem. |
| Permissions | The packed file is set to 644 so a non-root user can copy it off. |
| Cleanup | After pack or send the working directory is deleted. After review it stays. |
| Lock | One run per output directory (.<dir>.lock). A second run in the same place fails until the first finishes. |
<COMPANY>-v<VERSION>-appliance-<TIMESTAMP>/ summary.txt summary.html command line, company, comment, time zone, window, error table, system info errors/summary.txt error table only errors/plugin-crashes/<plugin>/<id>/ summary.txt plus the log the plugin died in errors/service-crashes/... service crash incidents with stacks and stats errors/disk-usage/ only if urgent DB purges happened (low disk) info/snapshot.log collection log and per-category elapsed time info/snapshot.properties version, nodeid, appliance|em, start, end, mem, endpoints, cpu_count, timezone info/misc/*.log output of the system commands below files/usr/local/forescout/... copied logs and config, original paths kept db/tables/ table dumps (hostinfo.dump.gz enables hostinfo diag later) debug-session/ only with --debug-session (traffic capture, traces)
uname -a, arch, dmesg, cat /root/.bash_history, lsof -n, lscpu, netstat -rn, netstat -naep, ip addr, ip route, ip link, ifconfig -a, df, fstool df, uptime, dmidecode, megacli -PDList -aAll, top -c -b -n1, ps auxww, ps -efww, rpm -qa --last, chkconfig --list, last -x, sar -r, ipmitool sel list, certutil -L -d etc/nss/db, fstool model, fstool hostid, fstool rms_license_info, fstool ird --all_appliances, fstool ird --all_irds, fstool fw status, fstool tech-support debug -l, fstool tech-support realmem, ls -liahFR of the main system directories. HA adds crm_mon -1r and cibadmin -Q; appliances add fstool npstats and fstool spt stats.
--anonymize when it leaves your control.This is the logic behind the plugin crash table in summary.txt. For each plugin launch event in the window that ended in a crash, it reads up to 1,000 lines (plugin_crash_last_lines) from the log the old process died in, then:
In plugin_stop_daemon_cb (a requested stop).is_restart_required:<n>: together with plugin restart required on this appliance (a configuration restart).Accepting connections on port N, or lines show shu_handle_message, Handling stop message stop_, need to restart or PLUGIN_EXIT_ON_PURPOSE.Timeout receiving message from CU (slow service) or Connection to timer server terminated.crash_patterns from the plugin's support.xml are checked against the last two log files.Out of memory!, or Plugin stalled if a log/Trace_cu* file shows Restarting plugin <id> because it is stalled within 60 seconds.plugin_stop_daemon_cb or is_restart_required is not counted here, while an Out of memory! death is counted and labelled. Run --health-check -p <plugin> -t 2d to get the tool's own verdict on each stop.--debug-session turns on extra tracing, waits while you reproduce the problem, turns it all back off, then collects a snapshot covering that window.
debug-session/traffic (disable with --define capture_enabled=0, change interface with --define capture_eth=ethX).--debug-session-trace key=level, skip with --no-debug-session-trace).--strace, attaches strace to the plugin processes.engine.record=true so the packet engine records raw traffic.-p plugin to debug level 4 (or its support.xml level, or --define plugin_debug_level=N) only if it is currently lower.10m, 1h) or for ENTER with i, then restores every level and property it changed.fstool tech-support -p sw --debug-session 15m --pack fstool tech-support -p dot1x --debug-session i --pack # reproduce, then press ENTER fstool tech-support -p sw --debug-session 10m --strace --define plugin_debug_level=5 --pack
tcpdump.pl replays raw engine data files through /bin/tcpdump inside the chroot /usr/local/forescout/etc/tcpdump as user nobody. It always prints with -n -e -X -s 5555 -tt and matches the filter with or without a VLAN tag.
fstool tcpdump -h <HOST_IP> -d <RECORD_DIR> # all files in a directory fstool tcpdump -h <HOST_IP> -p 1812 -r <FROM_EPOCH>-<TO_EPOCH> -d <RECORD_DIR> fstool tcpdump -f 'udp port 162' <DATA_FILE> <DATA_FILE> # specific files fstool tcpdump -v -h <HOST_IP> -d <RECORD_DIR> # show the tcpdump command used
The data file names are epochs, which is how -r from-to selects them.
diag takes a snapshot .tgz or extracted directory, restores its time zone, window, node ID and memory from info/snapshot.properties, and reruns the analysis into <DIR>.diag. It also runs the categories that only work remotely, such as stats_diag, and hostinfo_diag when db/tables/hostinfo.dump.gz is present.
fstool tech-support diag /tmp/<SNAPSHOT>.tgz fstool tech-support diag /tmp/<SNAPSHOT>.tgz -t "Oct 20 12:00" -t "Oct 20 14:00" # narrow the window
This is the way to read a snapshot pulled from another appliance: copy it to a lab appliance or Enterprise Manager on the same version and run diag there.
fstool tech-support --health-check --oneach # from the EM, all appliances, dashboard view fstool tech-support --health-check --oneach-all # plus the EM and the Recovery EM
--send and the send subcommand hand the directory to shoot_snapsend in support.ph, with company, description (comment plus summary) and an optional server. --server ADDRESS replaces the DNS name of the snapshot server; the help text gives <SNAPSHOT_SERVER_IP> as the example for when DNS does not resolve. The server value is rejected if it contains any of $ ` & ; < > { } % | * \ / ( ) !.
The upload protocol is implemented in support.ph, which is not part of this review, so pointing --server at your own machine is not a supported way to receive a snapshot: your server would have to speak Forescout's snapshot protocol. Pack the snapshot and push it yourself, as in section 12.
fstool tech-support -p sw -t 1d --send --company <COMPANY> --comment "Case <CASE>" fstool tech-support -p sw -t 1d --send --server <SNAPSHOT_SERVER_IP> --company <COMPANY> fstool tech-support send /tmp/<FILE>.tgz /usr/local/forescout/log/<FILE>.log
getent hosts <SNAPSHOT_SERVER_FQDN>, then curl -v https://<SNAPSHOT_SERVER_FQDN>. If it resolves to an internal address or times out, a firewall or proxy decision is needed, or use section 12.The appliance packs the snapshot, then pushes the file over HTTPS to a small receiver on your server. The receiver checks a shared token, writes the file with a timestamp and the sender's IP in its name, stores a SHA-256, and returns the hash so the appliance can confirm the copy is exact.
Files in this package: ts_receiver.py (receiver, Python 3 standard library only), ts_push.sh (appliance side), ts-receiver.service (systemd unit). Tested end to end: a 30 MB upload, a streamed upload, a wrong token rejected with 401, and the list endpoint.
sudo useradd -r -s /sbin/nologin tsrecv sudo mkdir -p /opt/ts_receiver /srv/techsupport sudo cp ts_receiver.py ts-receiver.service /opt/ts_receiver/ cd /opt/ts_receiver sudo openssl req -x509 -newkey rsa:2048 -nodes -days 825 -subj "/CN=ts-receiver" -keyout server.key -out server.crt openssl rand -hex 24 # this is <TOKEN> sudo sed -i "s/<TOKEN>/PASTE_TOKEN_HERE/" ts-receiver.service sudo chown -R tsrecv:tsrecv /opt/ts_receiver /srv/techsupport sudo chmod 600 /opt/ts_receiver/server.key sudo cp ts-receiver.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now ts-receiver sudo ufw allow from <APPLIANCE_SUBNET> to any port 8443 proto tcp # or firewall-cmd equivalent curl -k https://127.0.0.1:8443/health
Optional lock-down: add --allow <IP1>,<IP2> to only accept those source addresses, and --max-gb 20 to cap file size (20 GB default). Plain HTTP for a quick lab test: omit --cert and --key.
scp ts_push.sh root@<APPLIANCE>:/root/ && ssh root@<APPLIANCE> chmod +x /root/ts_push.sh /root/ts_push.sh -s https://<SERVER>:8443 -k <TOKEN> -p sw -t 1d -c "sw crash" -n <COMPANY> /root/ts_push.sh -s https://<SERVER>:8443 -k <TOKEN> -f /tmp/<EXISTING>.tgz # push one you already packed /root/ts_push.sh -s https://<SERVER>:8443 -k <TOKEN> -p sw -- --cores --dbtable hostinfo # extra tech-support flags
It ends with VERIFIED when the server's hash matches the local one, or CHECKSUM MISMATCH and exit 1.
F=/tmp/ts_$(hostname -s).tgz; fstool tech-support -p sw -t 1d --pack --packname $F --company <COMPANY> --comment "<COMMENT>" < /dev/null && sha256sum $F && curl -sk -H "X-Token: <TOKEN>" -T $F https://<SERVER>:8443/upload/$(basename $F)
Each appliance packs and pushes its own snapshot. setsid nohup keeps the job alive when the next oneach runs; write the token to a root-only file first so it does not sit in the process list.
fstool oneach -c 'umask 077; echo <TOKEN> > /root/.tstoken' fstool oneach -c 'F=/tmp/ts_$(hostname -s).tgz; setsid nohup sh -c "fstool tech-support -p sw -t 1d --pack --packname $F --company <COMPANY> --comment fleet < /dev/null && curl -sk -H X-Token:\$(cat /root/.tstoken) -T $F https://<SERVER>:8443/upload/\$(basename $F)" > /tmp/tspush.log 2>&1 < /dev/null & disown; echo started' fstool oneach -c 'tail -3 /tmp/tspush.log' # progress and the JSON receipt curl -sk -H "X-Token: <TOKEN>" https://<SERVER>:8443/list # what has arrived fstool oneach -c 'rm -f /root/.tstoken /tmp/ts_*.tgz /tmp/tspush.log' # clean up after
$ with a backslash wherever the value must expand on the appliance inside the nested sh -c, as shown with \$(cat ...).# SSH push to an upload-only account on your server (key on the appliance, account restricted to sftp) scp -i /root/.ssh/ts_upload /tmp/<SNAPSHOT>.tgz tsupload@<SERVER>:/srv/techsupport/ # Pull instead of push, from your server (only needs SSH into the appliance) scp root@<APPLIANCE>:/tmp/<SNAPSHOT>.tgz /srv/techsupport/ # Lab only, no encryption: listener then sender nc -l 9000 > /srv/techsupport/snap.tgz # on your server nc <SERVER> 9000 < /tmp/<SNAPSHOT>.tgz # on the appliance sha256sum /tmp/<SNAPSHOT>.tgz # compare both sides
<SERVER>:8443: curl -sk https://<SERVER>:8443/health returns {"ok": true}./tmp on the appliance has room for the pack; otherwise add --tmp /shared/<DIR> or --packname on a bigger filesystem.fstool tech-support --health-check # fast verdict, nothing packed fstool tech-support --health-check -p sw -t 2d # verdict on one plugin fstool tech-support -p sw -t 1d --pack # standard plugin case fstool tech-support -p sw -t 1d --pack --packname /tmp/<NAME>.tgz # exact output file fstool tech-support -p sw -t utc:<START> -t utc:<END> --pack # tight crash window fstool tech-support -p sw --debug-session 15m --pack # trace while reproducing fstool tech-support --hinfo <HOST_IP> --pack # one endpoint in detail fstool tech-support --dbtable <TABLE> --custom --include db --pack # one table only fstool tech-support --cores --pack # include core files fstool tech-support --anonymize --pack # anonymized .zip fstool tech-support --tmp /shared/ts --pack # /tmp too small fstool tech-support --list-items # categories on this box fstool tech-support version # plugin IDs fstool tech-support diag /tmp/<SNAPSHOT>.tgz # analyze offline fstool tech-support --health-check --oneach-all # every node from the EM fstool tech-support -p sw -t 1d --send --company <COMPANY> # to Forescout fstool tech-support -p sw -t 1d --send --server <SNAPSHOT_SERVER_IP> # DNS not resolving fstool tech-support send /tmp/<FILE> # send loose files to Forescout