Job Details

Splunk Engineer

Onboard new systems and data sources into the City's enterprise Splunk Cloud environment and developing dashboards, alerts, and analytics to improve operational visibility, security posture, and service reliability. This role is highly technical and requires strong expertise in Splunk administration, data onboarding, and SPL (Search Processing Language) development.

Key Responsibilities:

• Onboard new systems, logs, and data sources into Splunk, ensuring proper parsing, field extractions, CIM compliance, and data normalization.
• Configure and maintain forwarders, ingestion pipelines, and data routing.
• Build advanced dashboards, visualizations, and analytics for operational, security, and business use cases.
• Develop complex SPL queries, macros, lookups, and scheduled searches.
• Troubleshoot ingestion issues, search performance, and data quality problems.
• Partner with network, server, application, and security teams to define log requirements and actionable monitoring.

Minimum Qualifications:

• Experience administering and engineering Splunk Enterprise or Splunk Cloud in a medium-to-large environment.
• Strong proficiency with SPL for analytics and troubleshooting.
• Demonstrated experience onboarding new systems or applications into Splunk.
• Experience building dashboards with Splunk Dashboard Studio or Classic Editor.
• Knowledge of log ingestion formats (syslog, JSON, XML), data parsing, and field extraction.
• Understanding of IT infrastructure fundamentals (servers, networks, firewalls, cloud services).
• Experience with Linux command line and Splunk Universal/Heavy Forwarder management.

Preferred Qualifications:

• Experience with automation or scripting (Python, PowerShell).
• Experience with Enterprise Security (ES) or ITSI modules.
• Familiarity with indexer clustering, search head clustering, and distributed Splunk architectures.
• Experience implementing CIM compliance and data models.

Ideal Candidate:

• Strong analytical, troubleshooting, and visualization skills.
• Ability to work collaboratively with infrastructure, application, and security teams.
• Strong communication skills and the ability to translate technical findings into actionable insights.