We are seeking an experienced Databricks Architect to design, implement and optimise scalable enterprise data platforms, distributed processing solutions and AI/ML architectures using the Databricks platform.
Key Responsibilities

• Lead the end-to-end architecture and deployment of large-scale data platforms using Databricks, Apache Spark, Delta Lake and Unity Catalog.
• Design scalable batch and real-time data processing solutions and ETL/ELT pipelines.
• Develop medallion and lakehouse architecture patterns for enterprise data workloads.
• Architect secure and resilient Databricks environments across Azure, AWS or GCP.
• Establish data governance, access controls, lineage and compliance standards through Unity Catalog.
• Design integrations between Databricks and cloud storage, databases, APIs, BI platforms and enterprise applications.
• Define platform security standards covering identity, networking, encryption, secrets management and role-based access.
• Implement CI/CD, infrastructure-as-code and automated deployment practices for Databricks workloads.
• Optimise Spark jobs, clusters, SQL workloads and resource utilisation for performance and cost efficiency.
• Support the architecture and operationalisation of machine learning and generative AI solutions.
• Troubleshoot complex platform, integration and performance issues.
• Create reusable architecture patterns, standards and technical documentation.
• Collaborate with data engineers, AI/ML teams, security specialists, cloud engineers and business stakeholders.
• Provide technical leadership, architecture assurance and mentoring to engineering teams.
• Develop platform roadmaps and drive enterprise adoption of Databricks.

Mandatory Skills and Experience

• Extensive experience designing enterprise data platforms and lakehouse architectures.
• Strong hands-on Databricks architecture and implementation experience.
• Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
• Strong experience with Delta Lake and Unity Catalog.
• Experience designing ETL/ELT pipelines and distributed data-processing solutions.
• Strong cloud architecture experience across Azure, AWS or GCP.
• Experience with cloud storage, networking, identity and security services.
• Knowledge of data governance, lineage, access controls and regulatory compliance.
• Experience implementing CI/CD and infrastructure-as-code using tools such as Terraform, Azure DevOps, GitHub Actions or Jenkins.
• Strong understanding of SQL, Python and data-modelling principles.
• Experience integrating Databricks with enterprise data and analytics platforms.
• Strong stakeholder-management and technical-leadership capabilities.

Highly Desirable

• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
• Databricks Certified Generative AI Engineer or related architecture certification.
• Experience with MLflow, Feature Store, Model Serving and MLOps.
• Experience designing AI, machine-learning, GenAI or agentic AI solutions.
• Knowledge of streaming technologies such as Kafka, Structured Streaming or Event Hubs.
• Experience delivering regulated, government or large-enterprise data platforms.

To be considered for the role click the 'apply' button or for more information about this and other opportunities please contact Anshita Soni on 02 7257 8211 or email asoni@paxus.com.au and quote the above job reference number.

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.