Data Engineer

Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation.

Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks.

Key responsibilities

- Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata. - Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs. - Model and build raw-to-mart data layers that serve analytics and AI use cases. - Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG. - Set up and manage vector databases and knowledge repositories used by LLM agents. - Implement data quality rules, profiling, scoring outputs, and exception handling. - Register lineage and maintain source registries, version tracking, and refresh controls. - Design data stores for signals, findings, audit trails, and user feedback loops. - Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling. - Support SIT/UAT data validation, defect fixes, and production release activities.

Required skills

- Strong Python and SQL; solid data modeling (dimensional and normalized). - Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services. - Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar). - API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns. - Data quality frameworks and testing practices for pipelines. - Version control (Git) and CI/CD for data workloads.

Preferred skills

- RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar). - Unstructured content processing: text extraction, OCR, document parsing. - Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases). - Experience supporting LLM or agentic applications with grounded, traceable data. - Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling. - Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake).