We're looking for a seasoned Cloud Integration Lead with expertise in Databricks, Apache Spark, and cloud-based data integration. You'll have a strong technical background, excellent collaboration skills, and a passion for delivering high-quality solutions.
Requirements
- 5+ years of experience with Databricks, Apache Spark, and cloud-based data integration.
- Strong Technical expertise with cloud-based platforms, including AWS and or Azure cloud.
- Strong programming skills in languages like SQL, Python, Java, or Scala.
- 3+ years' experience with cloud-based data infrastructure and integration leveraging tools like S3, Airflow, EC2, AWS Glue, DynamoDB & Lambdas, Athena, AWS Code deploy, Azure Data Factory, or Google Cloud Dataflow.
- Experience with Jenkins and other CI/CD tools like GitLab CI/CD, CircleCI, etc.
- Experience with containerization using Docker and Kubernetes.
- Experience with infrastructure such as code using tools like Terraform or CloudFormation
- Experience with Agile development methodologies and version control systems like Git
- Experience with IT service management tools like ServiceNow, JIRA, etc.
- Data warehousing solutions, such as Amazon Redshift, Azure Synapse Analytics, or Google BigQuery will be a plus but not mandatory.
- Data science and machine learning concepts, including TensorFlow, PyTorch, or scikit-learn will be a plus but not mandatory.
- Strong technical background in computer science, software engineering, or a related field.
- Excellent collaboration, communication, and interpersonal skills.
- Experience with data governance, data quality, and data security principles.
- Ability to lead and mentor junior team members.
- AWS Certified Solutions Architect or AWS Certified Developer Associate or Azure Certified Solutions Architect certification.
Benefits
- Design and implement scalable data integration solutions using Databricks, Apache Spark, and cloud-based platforms.
- Develop and implement cloud-based data pipelines using Databricks, Nifi, AWS Glue, Azure Data Factory, or Google Cloud Dataflow.
- Collaborate with cross-functional teams to deliver high-quality solutions that meet business requirements.
- Develop and maintain technical standards, best practices, and documentation.
- Integrate various data sources, including on-premises and cloud-based systems, applications, and databases.
- Ensure data quality, integrity, and security throughout the integration process.
- Collaborate with data engineering, data science, and business stakeholders to understand requirements and deliver solutions.