We are seeking an experienced Platform Engineer for our insurance client. With strong hands-on capability across modern data platforms, distributed systems, and cloud environments, the ideal candidate will bring deep technical expertise, and the ability to operate and optimise large-scale data ecosystems. The successful incumbent will also have practical experience working with Databricks notebooks, clusters, jobs and working knowledge of cloud platforms (Azure preferred).
Essential Skills
- Strong experience managing and administering Cloudera Data Platform (CDP) clusters, including configuration, upgrades, security, and monitoring.
- Ability to troubleshoot platform-level issues across HDFS, YARN, Hive, Impala, and related ecosystem services.
- Hands-on experience managing Kafka clusters, including brokers, topics, partitions, consumer groups, schema registry, and connector management.
- Skilled in performance optimisation, scaling, and fault recovery scenarios.
- Strong knowledge of Iceberg tables, metadata management, partitioning, time-travel, and ACID capabilities.
- Experience implementing and supporting data Lakehouse architectures at scale.
- Expertise using Synapse Pipelines, Spark Pools, SQL Pools, and integration runtimes.
- Ability to optimise Synapse Spark jobs and SQL queries running on the platform.
- Proficiency developing automation scripts/playbooks for provisioning, configuration management, and infrastructure-as-code.
- Ability to manage reusable Terraform modules, state files, and CI/CD-driven deployments.
- Scripting – Bash / Python
- Ability to write robust Bash and Python scripts for automation, tooling, monitoring, and data processing.
- Strong debugging and shell-level troubleshooting skills.
- Solid ability to read, understand, debug, and optimise PySpark code.
- Extensive troubleshooting of Spark job failures, performance issues, memory errors, and cluster utilisation problems.
- Experience deploying, managing, and monitoring containerised workloads on Kubernetes clusters.
Desirable Skills
- Experience managing and administering IDMC components such as data integration, MDM, and data quality modules.
- Experience in administering Postgres databases in a cloud environment, including backups, maintenance, performance tuning, and replication.
To be considered for the role click the 'apply' button or for more information about this and other opportunities please contact Jennifer Bohm on 02 9464 5518 or email: jbohm@paxus.com.au and quote the above job reference number.
Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.
