MS or BS in Computer Science or Equivalent

6-10+ years of relevant experience

Strong Software Engineering Fundamentals & API Development

Proficiency in Data Structures & Algorithms: Critical for designing efficient systems that handle large-scale data. System Design: Ability to design scalable, fault-tolerant systems and services. Coding Skills: Expertise in languages like Java, Python, or Scala, especially for backend systems development. RESTful Services: Proven experience in designing and building robust APIs for data access and integration. GraphQL: Knowledge of modern data access technologies for flexible querying. Microservices Architecture: Experience with creating microservices that handle different aspects of data management.

 

Data Architecture and Design Patterns

Database Design: Strong knowledge of both relational (SQL) and non-relational (NoSQL) databases like Oracle, MongoDB, Cassandra, or DynamoDB. Data Modeling: Ability to design and manage data models for efficient storage and retrieval. Data Warehousing: Experience with data warehouses and ETL processes. Data Access Controls: Experience with role-based access control (RBAC) and encryption techniques to secure sensitive data. Metadata Management: Familiarity with tools and processes for tracking data lineage, metadata catalogs (e.g., Apache Atlas, DataHub). Ability to handle complex data-related challenges, from dealing with incomplete or inconsistent data to optimizing performance. Strong analytical thinking to derive insights from data and build solutions that improve platform performance.

 

Data Pipeline and Orchestration

ETL/ELT Tools: Experience with data pipeline orchestration tools like Airflow, Prefect, or Dagster. Automation & CI/CD: Familiarity with setting up CI/CD pipelines for data infrastructure. Distributed Systems: Experience with technologies like Hadoop, Spark, Kafka, and Flink for managing large-scale data processing pipelines. Data Streaming: Familiarity with real-time data processing frameworks (e.g., Apache Kafka, Pulsar, or RabbitMQ). Cloud Platforms: Hands-on experience with cloud-based data services (AWS, GCP, Azure), including data storage (S3, GCS) and data analytics (EMR, Dataproc).

 

Leadership and Collaboration

Team Leadership: Ability to mentor junior engineers and guide them in building robust, scalable systems. Cross-Functional Collaboration: Experience working closely with data scientists, analysts, and other engineers to deliver on the platform’s objectives. Stakeholder Management: Strong communication skills for presenting technical decisions to non-technical stakeholders.

Career Level - IC4