Data Platform Engineer Jobs in San Francisco, CA

Senior Data Platform Engineer

Location - US Remote. Candidate to be based out of US.

Mandatory Skills

• designing, optimizing, and managing enterprise-grade data ecosystems.

• mastery of database architecture, performance tuning, and transaction management,

• hands-on experience in cloud-native platforms, big data frameworks, and advanced analytics pipelines.
  

The ideal candidate will be comfortable working across distributed systems, modern data engineering tools, and collaborating with architects, DevOps, and business teams to deliver scalable, secure, and high-performing solutions.

KEY RESPONSIBILITIES

Database & System Architecture

• Design and implement robust, scalable, and secure database architectures for transactional and analytical workloads.
• Define partitioning strategies, indexing, and normalization for optimal performance and maintainability.

Performance Tuning & Optimization

• Conduct advanced query optimization and execution plan analysis.
• Implement proactive monitoring and troubleshooting for high-volume, mission-critical systems.

Cloud & Distributed Systems

Azure Service Fabric:

• Deploy and manage microservices-based applications on Azure Service Fabric clusters.
• Ensure high availability, fault tolerance, and scalability of distributed services.
• Optimize communication patterns and stateful/stateless service design for data-intensive workloads.

Modern Data Engineering & Analytics

Azure Data Factory (ADF):

• Design and orchestrate complex ETL/ELT pipelines for batch and streaming data.
• Integrate diverse data sources into centralized data platforms.

Azure Databricks & Spark:

• Build and optimize big data processing workflows using Apache Spark on Databricks.
• Implement Delta Lake for ACID transactions on large-scale data lakes.
• Develop scalable solutions for machine learning and advanced analytics.

Delta Tables:

• Manage versioned data storage for reliability and reproducibility.
• Optimize schema evolution and data compaction strategies.

Power BI (PBI):

• Collaborate with BI teams to design semantic models and optimize data for reporting.
• Ensure data pipelines deliver clean, consistent, and performant datasets for visualization.

Collaboration & Leadership

• Work closely with architects, DevOps, and business stakeholders to align technical solutions with strategic objectives.
• Provide mentorship and technical guidance to junior engineers and developers.

REQUIRED SKILLS & QUALIFICATIONS

Database Mastery

• Advanced SQL, query optimization, partitioning, and transaction handling.

Performance Tuning

• Proven ability to diagnose and resolve complex performance bottlenecks.

Azure Expertise

• Hands-on experience with Azure Service Fabric, Azure Data Factory, Azure Databricks, and Azure Storage.

Big Data & Analytics

• Strong knowledge of Spark, Delta Lake, and distributed data processing.

Data Visualization

• Familiarity with Power BI and data modeling best practices.

Troubleshooting & Optimization

• Ability to resolve issues across multi-tiered systems under pressure.

Cross-Functional Collaboration

• Excellent communication and stakeholder management skills.

PREFERRED QUALIFICATIONS

• Experience with CI/CD pipelines for data solutions.
• Familiarity with data governance, security, and compliance frameworks.
• Exposure to machine learning workflows and real-time streaming architectures.