Principal Consultant

Key Responsibilities

• Design, develop, and maintain data processing applications using Python and Cloudera technologies.
• Build and optimize ETL pipelines and large-scale data ingestion frameworks.
• Develop and maintain Spark/PySpark applications for distributed data processing.
• Work with Hadoop ecosystem components including Hive, HDFS, and Spark.
• Develop web-based tools and internal applications using Django.
• Implement data governance, security, and access controls using Ranger and Atlas.
• Automate operational tasks using Shell scripting and Linux tools.
• Design and manage data storage solutions using Parquet, ORC, JSON, and other data formats.
• Troubleshoot and resolve performance, scalability, and production issues.
• Collaborate with Data Engineers, Data Scientists, Architects, and DevOps teams to deliver enterprise data solutions.
• Participate in code reviews, design discussions, and best-practice implementation.
• Create technical documentation and support knowledge-sharing initiatives.

Requirements:

• Strong hands-on experience in Python development.
• Experience with Cloudera Data Platform (CDP) and Cloudera Data Science Workbench (CDSW).
• Strong understanding of Hadoop ecosystem (HDFS, Hive, YARN).
• Experience in Spark/PySpark application development and optimization.
• Strong SQL programming and query optimization skills.