Job Description
Role & responsibilities
Domain: Endpoint Device Insights & High-Throughput Telemetry Networks
## Job Purpose
We are seeking a high-caliber, delivery-focused Principal Systems Architect to spearhead the core data processing and analytics ecosystem for our next-generation endpoint insights platform. In this role, you will bridge the gap between low-latency IoT/device edge ingestion and massive-scale data analytics pipelines.
You will own the system design, end-to-end telemetry architecture, and technical execution across both hybrid cloud and strictly isolated on-premise environments. This role demands a hands-on builder who lives in code, masters real-time data engines, and architectural frameworks like the Medallion design pattern to guarantee maximum scale, resilience, and operational availability.
## Core Architectural Mission: The 7R Drivers
You will actively enforce the 7R software architecture drivers across every micro-service and storage engine you design:
- Resiliency: Designing self-healing infrastructure, transient fault handling, and circuit breakers.
- Reliability: Ensuring complete data integrity from source edge ingestion to analytical reporting.
- Responsiveness: Maintaining low-latency handling of incoming telemetry streams and real-time push events.
- Robustness: Building deep error-defensive mechanisms against unstable device inputs and edge data mutation.
- Recoverability: Formulating granular disaster recovery patterns across stateful processing and high-scale storage nodes.
- Resource Efficiency: Optimizing hardware footprints across on-prem bare-metal installations and cloud computing environments.
- Readiness (Availability): Designing active-active multi-region networks targeting zero single-points-of-failure.
## Key Responsibilities
- High-Throughput Ingestion Design: Architect and refine real-time endpoint ingestion systems handling millions of events using lightweight message protocols like MQTT.
- Data Lakehouse & Warehouse Optimization: Design, scale, and maintain our dual-engine analytics engine using ClickHouse (for massive, high-speed trace and log aggregation via MergeTree families) and PostgreSQL (for relational configuration, dynamic state management, and asset metadata).
- Medallion Pipeline Engineering: Implement multi-tier data processing workflows leveraging the Medallion Architecture (Bronze ingestion, Silver clean telemetry processing, and Gold enterprise insight aggregates).
- Hybrid & On-Prem Deployment Engineering: Architect secure, containerized software systems that run seamlessly in cloud infrastructure and strictly firewalled, air-gapped on-premise environments.
- Bi-directional Communication: Design and scale reliable system integrations including real-time outbound Push Notifications and downstream micro-services hooks directly to source endpoint ecosystems.
- AI-Driven Programming & Engineering: Actively accelerate development velocity by integrating advanced LLM/AI workflows into code generation, comprehensive test automation, optimization pipelines, and operational telemetry parsing.
## Technical Skill Requirements
### Primary Technical Stack (Must Have)
- Expert Python Systems Development: Extensive background in building high-concurrency, asynchronous systems (Asyncio, Celery, FastAPI, or equivalent) for data routing and edge ingestion.
- Message Broking: Deep engineering experience with MQTT brokers (e.g., EMQX, Mosquitto, HiveMQ) handling millions of concurrent persistent device connections.
- Advanced Data Store Architecture:
- Proven experience tuning and writing high-performance layouts in ClickHouse (Materialized Views, Skip Indexes, optimized table engines).
- Expert knowledge of PostgreSQL cluster, query optimisation, data pipelines, complex indexing, state storage, and relational modeling at scale.
- Architectural Mastery: Demonstrated experience implementing Medallion Data Layouts or distributed Event-Driven Microservices Architectures.
### Secondary & Ecosystem Experience
- Platform Operations: Deep experience handling both public cloud platforms and physical On-Prem deployments(Bare metal, Kubernetes/K3s clusters, network routing constraints).
- System Integrations: Experience architecting clean Webhooks, enterprise REST/gRPC API structures, and complex system orchestration hooks.
- Real-time Alerting: Designing scalable pub-sub channels for targeted outbound notifications and mobile/desktop event delivery networks.
- Modern Tooling: Comfort utilizing modern AI coding extensions, advanced CI/CD tooling, Docker/Podman container runtimes, and deep system instrumentation hooks for tracking application performance.
## Behavioral & Delivery Requirements
- Hands-on Delivery Champion: This is not a "whiteboard-only" consulting role. You are expected to code, profile benchmarks, write technical blueprints, and pair-engineer high-risk segments alongside developers.
- Pragmatic Problem Solver: Balancing the purity of theoretical architecture with the immediate delivery goals of enterprise SaaS deployments.
- Systems Thinker: Ability to analyze how a micro-optimization in a Python agent affects disk I/O metrics across a ClickHouse cluster.
Preferred candidate profile