The Senior Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and cloud-based solutions that power our internal systems and business intelligence capabilities. This role sits at the intersection of data engineering, backend development, and cloud architecture, working closely with cross-functional teams across Sales, Accounting, and Operations.
Key Responsibilities
- Design, develop, and maintain end-to-end data pipelines on Databricks (Unity Catalog, Delta Lake, PySpark)
- Architect and implement cloud solutions on Azure: Container Apps, Container Registry, Blob Storage, Azure SQL, and Azure Functions
- Develop APIs and backend services using Python (FastAPI) for integration of internal and external systems
- Optimize pipeline performance: reduce runtimes, implement checkpoint strategies, and enable parallel processing
- Model data, design schemas, and manage relational database evolution
- Integrate with third-party systems via APIs and webhooks (TMS, pricing systems, logistics platforms)
- Develop dashboards and internal tools for business areas including Sales, Accounting, and Operations
- Write and maintain technical documentation for architectures and implemented solutions
- Provide technical support and troubleshooting for production systems
- Languages: Python (FastAPI, pandas, PySpark), SQL, JavaScript
- Cloud: Microsoft Azure (Container Apps, Container Registry, Blob Storage, SQL Database, Functions, Function Apps)
- Data Platform: Databricks, Unity Catalog, Delta Lake