ETL/Data Engineer - Tech Lead
- New York, NY, USA New York NY US
- Permanent, Full time
- S&P Global
- 23 Jun 18 2018-06-23
ETL/Data Engineer - Tech Lead
The ETL Tech Lead will design, develop and deliver strategic data-centric applications leveraging the firm's next generation ETL technologies with a strong focus on tools like Informatica etc. These solutions will be architected in alignment with the underlying S&P Global technology infrastructure, as well as the foundational Data Services' framework.
This role is a hands-on technical role.
• Provide leadership for all stages for the software development /maintenance for the Business Intelligence application project portfolio. This includes oversight for the design, specification, ongoing maintenance and roadmap of this project portfolio:
• Development of Mappings, data flows
• Diagnosing issues.
• Data structure optimization and integration
• Automation, deployment, scheduling & distribution of feeds
• Auditing & value realization (usage metrics)
• Collaborating with on/off-shore resources
• Knowledge about Data Analytics and ability to provide end to end solutions
• Ability to take Initiative and Proactively address problems
• Partner with enterprise architects to define and ensure proposed Business Intelligence solutions adhere to enterprise reference architecture.
• Design robust data centric Business Intelligence solutions that consider technology from a development, operations, business, and vendor management perspective.
• Actively participates and represents Business Intelligence Solutions team in meetings and facilitates cross-functional team collaboration.
• Create and deliver project communications and presentations to relevant stakeholders
• Aligns application systems design with the business voice of the customer and articulates trade-offs and alternative options.
• Deliver the optimal mix of approach, process, and technology that best accomplish the goals of an application/project while adhering to enterprise reference architecture.
- 5+ years "hands-on" experience designing Data Integration solutions with the latest ETL and Analytic Tools.
- 5+ years "hands-on" experience using Oracle PL/SQL, stored procedures, SQL optimization skills
- Proven experience with data integration & Data Analytic tool set(s). Data warehousing skill set is a plus.
- Very strong data modeling skills
- Experience with Taxonomy (XSD) and generating XML files is a plus.
- Must be "hands-on" as well as be able to manage other resources in completing a project;
- Experience with software development lifecycle methodologies (Agile preferred)
- Strong communication, presentation and interpersonal skills are essential
- Technical leadership and mentoring skills
- Bachelor's degree and/or Masters degree in computer science or related fields
- Ability to collaborate with business and technology teams to create practical, robust and scalable architectures and solutions meeting the business and technology goals/strategy of of the organization
- Hands on with Python ,with prior experience in handling ETL/ELT workload using Python Scripts.
- Hands on with loading and manipulating large data sets using Spark/PySpark and SparkSQL.
- Prior Experience with consuming / Persisting data from and to Relational databases, S3 and Redshift using Python/PySpark.
- Experience with various AWS EMR components ( Hive /HDFS/Spark/ EMFRS/Scoop) handling very large data sets in a large Data Lake setup.
- Good Understanding of other AWS services like S3 ,EC2 , IAM , RDS Experience with Orchestration and Data Pipeline like AWS Step functions/Data Pipeline/Glue