AVP, Senior Data Engineer, Group Consumer Banking Technology, Technology & Operations
Group Technology and Operations (T&O) enables and empowers the bank with an efficient, nimble, and resilient infrastructure through a strategic focus on productivity, quality & control, technology, people capability and innovation. In Group T&O, we manage most of the Bank's operational processes and inspire to delight our business partners through our multiple banking delivery channels.
As a Senior Data Engineer, you'll be a specialist defining Data design, architecture, and strategy for C2MA platform projects, ensuring effective adoption of DBS enterprise Big Data stack and help us discover the information hidden in vast amounts of data. You'll help us make smarter decisions to deliver even better products and apply data mining techniques and statistical analysis to build high quality prediction systems integrated with our products. Responsibilities
- Design and implement key components for highly scalable, distributed data collection and analysis system built for handling petabytes of data in the cloud.
- Move architecture and implementation through the development pipeline, from research to deployment
- Work with architects from other divisions contributing to this analytics system and mentor team members on best practices in backend infrastructure and distributed computing topics.
- Analyse source data and data flows, working with structured and unstructured data.
- Manipulate high-volume, high-dimensionality data from varying sources to highlight patterns, anomalies, relationships and trends
- Analyse and visualize diverse sources of data, interpret results in the business context and report results clearly and concisely.
- Work side-by-side with product managers, software engineers, and designers in designing experiments and minimum viable products.
- Enhance data collection procedures that is relevant for building analytic systems.
- Aid project teams to discover data sources, get access to them, import them, clean them up, and make them "model-ready". You need to be willing and able to do your own ETL.
- Ability to install, analyse and evaluate different vendor products and tools catering to different areas of data work (Data exploration, Data visualization and MLOps) in the DBS Bigdata ecosystem and enable the business and Data scientists.
- Design and build API's to fetch and processing(request based) of the batch curated data.
- 6+ years of Experience in one or more areas of big data and machine learning
- Able to work with loosely defined requirements and exercise your analytical skills to clarify questions, share your approach and build/test elegant solutions in weekly sprint/release cycles.
- Development experience in Java/Scala and pride in producing clean, maintainable code
- Experience creating pipelines to analyze data, extracted features and prep the ML model
- Independence and self-reliance while being a pro-active team player with excellent communication skills, able to lead and mentor distributed teams.
- Professional experience building enterprise Big Data Applications using Spark, Hadoop, Hive, Presto, & Airflow.
- Hands-on development of Data applications using Spark Execution Framework, Spark SQL, Scala/Python(pyspark)
- Experience with distributed databases, such as MongoDB Aerospike, and the key issues affecting their performance and reliability.
- Experience using high-throughput, distributed message queueing systems such as Kafka.
- Familiarity with operational technologies, including Docker (required), Chef, Puppet, Zookeeper, Terraform, and Ansible (preferred).
- An ability to periodically deploy systems to on-prem environments.
- Mastery of key development tools such as GIT, and familiarity with collaboration tools such as Jira and Confluence or similar tools.
- Experience with Teradata SQL, Exadata SQL, T-SQL.
- Experience in migrating SQL from traditional RDBMS to Spark and Bigdata technologies
- In-depth knowledge of database internals and Spark SQL Catalyst engine
- Performance optimization in distributed compute environment (Spark & Hadoop).
- Experience/Exposure to SAS programming fundamentals
- Exposure to Data exploration tools like DataIQ and onboarding them into enterprise environments
- Handling of Data visualization tools like (Tableau, QlikView)
- Working experience with web-services (REST, SOAP) and/or experience in Microservices will be add on.
- Good with Architecture and has good understanding and in-depth knowledge on distributed systems
- Good knowledge of OOPs, data structure, and algorithm
We offer a competitive salary and benefits package and the professional advantages of a dynamic environment that supports your development and recognises your achievements.