Senior Big Data Engineer
Clearing, Markets & Issuer Services Technology (CMIST)
is responsible for applicationdevelopment and support for critical business systems including Repo Edge(collateral management), Enterprise Payment Hub (multi-currency paymentprocessing), and Broker Dealer Clearance (securities clearing), along withapproximately 350 other applications used by the following high-prioritybusiness services and their clients. Clearance andCollateral Technology (CCT)
withinCMIST builds clearance and collateral management platforms toservice BNY Mellon's broker-dealer clients and is the sole provider for GovernmentSecurities Clearing Services. Considering the core nature of the business andthe dominant market share, high-performance and resiliency are key pillars ofthe technology architecture. The group is currently focused on RepoModernization, which will pave the way to the Future of Collateral for thebusiness - providing strengthened platform resiliency and new tradecapabilities within a unified system. All CCT applications are built forreal-time operations aided by a data warehouse for reporting and businessinsights. This is a 100% hands-on role, and doesn't involve any teammanagement responsibilities Responsibilities:
- Develop, deploy, and maintain scalable/optimal data pipeline and the supporting data platforms and infrastructure
- Standardize all data pipeline processes, develop reusable modules (auditing, logging, traceability), integrate with DevOps pipeline
- Recommend improvements to existing data integration architecture and implementation of standard frameworks that automate manual processes, improve overall scalability, increase reliability, and improve quality
- Define the right data models working with ML engineers, data scientists, and reporting/analytics developers - optimizing data storage, improving data accessibility, and making them re-usable
Consults with internal business groups to provide highly advanced application software development services or technical support. Consults to IT and business managers on issues, questions and problems related to his/her area of expertise. Accountable for resolving highly complex problems that have impact on the corporation's ability to conduct business. Uses in-depth knowledge of various technologies to establish and influence implementation of broad-based technology strategy for large IT Divisions. Stays abreast of emerging technologies and identifies potential use of new and existing technology within lines of business by participating in industry-wide conferences and research and by using in-depth knowledge of various business areas. Educates business leaders and prepares presentations for the business on IT strategies. Advocates for innovative, creative technology solutions. Contributes to the achievement of Application Development objectives. Qualifications
BNY Mellon is an Equal Employment Opportunity/Affirmative Action Employer.
- BS degree (Computer Science, Math, Physics, Engineering). MS / PhD preferred.
- 12+ years of experience in software development required
- 5+ years overall hands-on experience in building and maintaining production grade data pipelines, developed in Python or Java, that support critical business functions, reporting, and analytics
- 3+ years overall hands-on experience in Python
- Very strong SQL skills with multiple years of experience in developing and maintaining complex SQLs across different standard and big data technologies (preferred)
- Extensive experience with big data platforms, technologies, and tools and technologies (Cloudera Hadoop, Spark, Hive, Impala, Dremio, Google BigQuery, ECS, etc.) - understanding their architecture, infrastructure requirements, troubleshooting tricks, integration with other platforms, and automation of processes
- Should have experience in handling large volume/size (billion / > 100 TB), wide data sets (> 200) and velocity (
- Hands-on experience around compute and storage in any one of the public cloud platforms such as Google Cloud (preferred), AWS, or Azure is highly preferred.
- Experience in the securities or financial services industry is a plus.
Minorities/Females/Individuals With Disabilities/Protected Veterans. Our ambition is to build the best global team - one that is representative and inclusive of the diverse talent, clients and communities we work with and serve - and to empower our team to do their best work. We support wellbeing and a balanced life, and offer a range of family-friendly, inclusive employment policies and employee forums. Primary Location:
United States-New York-New York Internal Jobcode:
Information Technology Organization:
Clearing Markets ISS Svcs Tech-HR16624 Requisition Number: