Databricks Infrastructure and DevOps Engineer

Job Role Name : Databricks Infrastructure and DevOps Engineer Key Responsibilities Infrastructure Management: -Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment

• Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB) -Configure and Optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks
• Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements CI/CD Pipeline Management: -Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem -Configure GitLab for version control and automated deployment workflows -Implement code quality gates using SonarQube for static code analysis
• Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment -Oversee dependency scanning and management using Nexus IQ -Coordinate with Cloud SCAPE for security compliance scanning and remediation Security and Compliance: -Implement and maintain Automated Baseline Log Review (ABLR) systems -Coordinate with Gov Tech Cyber Security Operations Centre (GCSOC) for security monitoring
• Manage Central Accounts Management (CAM) integration for user access control -Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT)findings.
• Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements -Implement and maintain Vulnerability Management System (VMS) Platform Operations: -Monitor system performance and implement optimization strategies
• Manage secrets and encryption key rotation for data protection -Configure email notifications for data processing status and error alerts -Implement and maintain data retention policies -Support SaaS whitelisting processes and approvals -Coordinate System Security Acceptance Testing (SSAT) activities Collaboration and Support:
• Work closely with Data Engineers, Data Analysts and security technical team. -Support multiple data marts -Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines -Collaborate with various government divisions and stakeholders -Support training environment setup for user onboarding and skill development Qualifications Technical Skills:
• 3+years of experience with Databricks platform administration and optimization
• Strong expertise in AWS services, particularly in GCC environment -Proficiency in Infrastructure-as-Code tools, preferably Terraform -Experience with GitLab CI/CD pipeline development and management -Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ) -Understanding of government security frameworks and compliance requirements -Experience with containerization and orchestration technologies
• Strong scripting skills in Python, Bash, or PowerShell Security and Compliance: -Experience with government security clearance processes -Knowledge of data classification and handling procedures (Sensitive High) -Understanding of encryption mechanisms and key management -Familiarity with vulnerability management and remediation processes -Experience with security monitoring and incident response procedures Certifications(Preferred): -Databricks Certified Associate or higher
• AWS Certified DevOps Engineer Required Experience -Minimum 3 years of DevOps/Infrastructure engineering experience -Experience working in government environments, esp in GCC. -Experience in agile development methodologies