3352 Data Engineer (Expert)

ROLE & RESPONSIBILITIES:

• Design, build and operate operational data integration solutions focused on making enterprise data available, trusted and consumable.
• Ingest and stream data from Zero Trust data sources (CMDB, identity systems, security tooling) into operational pipelines rather than analytics stores.
• Build and maintain Kafka topics, producers/consumers, connectors and streaming applications to enable realtime flows.
• Implement API-based integrations, webhook listeners and file ingestion solutions to capture operational events.
• Develop robust ETL/ELT and stream processing logic to normalise, correlate and enrich data across sources.
• Ensure pipelines are idempotent and resilient (retries, replay, backpressure and compensation strategies).
• Apply data modelling and metadata practices to ensure consistent schemas and discoverability.
• Implement and monitor data quality checks, anomaly detection and validation rules; drive remediation where needed.
• Maintain data lineage, access control and security practices to meet governance and compliance requirements.
• Collaborate with automation and orchestration teams to ensure streamed operational data feeds automation workflows and runbooks.
• Own pipeline alerting, runbooks, and participate in incident response/on-call rotations
• Troubleshoot production issues, perform root cause analysis and implement long-term reliability improvements.
• Build testable pipelines with unit/integration tests, contract tests and CI/CD deployment pipelines for streaming workloads
• Data Quality Gates: enforce automated quality checks early in pipelines (validation, schema checks, anomaly detection) and block or quarantine bad data.
• Performance & Cost Awareness: implement efficient partitioning, batching and resource usage patterns while tracking cost and throughput trade-offs.
• Security & Privacy by Design: apply least-privilege, secrets management, encryption in transit and at rest, and data masking/anonymisation where required.
• Drive architectural decisions and define integration best practices

• Extensive hands-on experience (typically 6+ years) in data engineering, integration or streaming roles with demonstrable production experience.
• Proven track record building and operating streaming platforms (Kafka) and API-based integrations, with strong Python/Java and enterprise databases and query languages skills.
• Strong analytical thinking, curiosity about data, attention to detail, structured problem solving and ownership - able to figure things out and drive topics to completion.