Senior Software Engineer

As a Senior Software Engineer focused on DataPelago data processing customer success, you will play a critical role in developing workload assets and automation tools that support product validation and customer support. Your work will ensure that customers can successfully deploy and operate DataPelago’s data processing products in production environments. By effectively integrating AI with insights gained from both product validation and support, you will help proactively prevent bugs and issues, while significantly reducing the time required to diagnose, mitigate, and resolve problems.

Key Responsibilities

- Design and develop synthetic workload queries and datasets that simulate real-world environments and workload characteristics for testing DataPelago products.
- Build analysis and diagnostic tools to interpret logs from DataPelago software, Apache Spark, and related deployment environments, facilitating root cause identification of functional and performance anomalies along with potential solutions.
- Participate in product support rotations by addressing or escalating customer issues promptly and effectively.
- Conduct comprehensive end-to-end product validation by collaborating with cross-functional teams to define, review, and execute detailed test plans.
- Analyze testing artifacts and customer-reported issues to uncover opportunities for configuration tuning, recommend best practices for product usage, and identify product bugs.
- Utilize automation and AI techniques throughout the product validation and support lifecycle to enhance efficiency and accuracy.
- Prepare detailed bug reports, root cause analysis (RCA) documents, configuration guides, and best practice documentation to support internal teams and customers.

Required Qualifications

- A minimum of 8 years of professional experience in the development, validation, and support of enterprise data processing software products.
- At least 4 years of direct experience providing customer support to external clients.
- Strong expertise in supporting Apache Spark pipelines for data engineering and analytics, including running Spark jobs from various clients, resource provisioning, parameter tuning, and log analysis using tools like Spark History Server and Spark UI.
- Proficiency in SQL and Python programming languages.
- Demonstrated experience in testing data products and implementing test automation frameworks.
- Solid understanding of deployment and operational aspects of data processing products and pipelines in cloud environments, including SaaS, virtual machines, containers, and serverless architectures.
- Familiarity with cloud platforms such as Google Cloud Platform (GCP) and Amazon Web Services (AWS), particularly their Apache Spark-related services.
- Excellent customer engagement and communication skills to effectively collaborate with clients and internal teams.

Preferred Qualifications and Benefits

- Experience ranging from 8 to 12 years in relevant roles is preferred, highlighting a strong background in enterprise data processing software.
- Exposure to multiple cloud environments and advanced automation techniques will be advantageous.
- Opportunity to work in a dynamic environment where AI-driven solutions are integral to product success and customer satisfaction.

This role offers the chance to contribute to cutting-edge data processing technologies while engaging directly with customers to ensure their operational success. If you bring a blend of technical expertise, customer focus, and innovative thinking, you will thrive in this position.