Senior Engineer – Machine Learning Platforms
Position Summary:
We are looking for a talented Senior Engineer to build and manage self-serve platforms focused on real-time ML deployment and advanced data engineering. This role combines expertise in cloud-native platform engineering, data pipeline creation, and seamless deployment of machine learning models at scale.
Responsibilities:
- Design and develop scalable microservices-based platforms using Kubernetes, Docker, and Python (FastAPI) for managing ML workflows and data pipelines.
- Architect and implement real-time ML inference platforms using AWS SageMaker, Databricks, and ensure model versioning, monitoring, and lifecycle management.
- Build and enhance ETL/ELT pipelines using PySpark and Pandas, and manage feature stores for high-quality data.
- Design distributed data pipelines, integrating DynamoDB, PostgreSQL, MariaDB, and other databases.
- Implement data lakes and warehouses to support analytics and ML workflows.
- Design, optimize, and maintain CI/CD pipelines with Jenkins and GitHub Actions for continuous deployment and testing.
- Automate monitoring and validation to ensure consistent data quality.
- Collaborate with cross-functional teams to understand business requirements and deliver integrated solutions.
- Maintain comprehensive technical documentation for platforms and workflows.
Required Skills & Qualifications:
- 5+ years of experience in platform engineering, data engineering, or DevOps roles.
- Proficient in Python, PySpark, FastAPI, Kubernetes, and Docker.
- Strong experience with AWS services, including SageMaker, Lambda, DynamoDB, and EC2.
- Expertise in managing distributed data pipelines with Databricks and PostgreSQL.
- Familiarity with CI/CD tools like Jenkins, GitHub Actions, and monitoring tools like New Relic.