Senior Big Data Engineer
Clearing, Markets & Issuer Services Technology (CMIST)
is responsible for application development and support for more than 350 critical business systems including Repo Edge (collateral management), Enterprise Payment Hub (multi-currency payment processing), and Broker Dealer Clearance (securities clearing). Clearance and Collateral Technology (CCT)
within CMIST builds high-performance and resilient clearance and collateral management platforms to service BNY Mellon's broker-dealer clients and is the sole provider for Government Securities Clearing Services. The group is currently focused on Repo Modernization, which will pave the way to the Future of Collateral for the business - providing strengthened platform resiliency and new trade capabilities within a unified system. All CCT applications are built for real-time operations aided by a data warehouse for reporting and business insights. This is a 100% hands-on role, and doesn't involve any team management responsibilities Responsibilities:
- Develop, deploy, and maintain scalable/optimal data pipeline and the supporting data platforms and infrastructure
- Standardize all data pipeline processes, develop reusable modules (auditing, logging, traceability), integrate with DevOps pipeline
- Recommend improvements to existing data integration architecture and implementation of standard frameworks that automate manual processes, improve overall scalability, increase reliability, and improve quality
- Define the right data models working with ML engineers, data scientists, and reporting/analytics developers - optimizing data storage, improving data accessibility, and making them re-usable
Sr. Principal Developer->> Consults with internal business groups to provide highly advanced application software development services or technical support. Consults to IT and business managers on issues, questions and problems related to his/her area of expertise. Accountable for resolving highly complex problems that have impact on the corporation's ability to conduct business. Uses in-depth knowledge of various technologies to establish and influence implementation of broad-based technology strategy for large IT Divisions. Stays abreast of emerging technologies and identifies potential use of new and existing technology within lines of business by participating in industry-wide conferences and research and by using in-depth knowledge of various business areas. Educates business leaders and prepares presentations for the business on IT strategies. Advocates for innovative, creative technology solutions. Contributes to the achievement of Application Development objectives. Qualifications
- BS degree (Computer Science, Math, Physics, Engineering). MS / PhD preferred.
- 12+ years of experience in software development required
- 5+ years overall hands-on experience in building and maintaining production grade data pipelines, developed in Python or Java, that support critical business functions, reporting, and analytics
- 3+ years overall hands-on experience in Python
- Very strong SQL skills with multiple years of experience in developing and maintaining complex SQLs across different standard and big data technologies (preferred)
- Extensive experience with big data platforms, technologies, and tools and technologies (Cloudera Hadoop, Spark, Hive, Impala, Dremio, Google BigQuery, ECS, etc.) - understanding their architecture, infrastructure requirements, troubleshooting tricks, integration with other platforms, and automation of processes
- Should have experience in handling large volume/size (billion / > 100 TB), wide data sets (> 200) and velocity (
- Hands-on experience around compute and storage in any one of the public cloud platforms such as Google Cloud (preferred), AWS, or Azure is highly preferred.
- Experience in the securities or financial services industry is a plus.
. BNY Mellon is an Equal Employment Opportunity/Affirmative Action Employer.
Minorities/Females/Individuals With Disabilities/Protected Veterans. Our ambition is to build the best global team - one that is representative and inclusive of the diverse talent, clients and communities we work with and serve - and to empower our team to do their best work. We support wellbeing and a balanced life, and offer a range of family-friendly, inclusive employment policies and employee forums. Primary Location:
United States-New Jersey-Jersey City Internal Jobcode:
Information Technology Organization:
Clearing Markets ISS Svcs Tech-HR16624 Requisition Number: