Modernizing Data Ingestion for Green Energy AI

Overview

JashDS modernized and automated data ingestion for a green energy AI solutions provider by developing a pipeline_builder library, reducing pipeline creation time by 40%, and improving data accessibility for 40+ utility sources.

About the Client

A provider of AI-based Green Energy solutions serving 40+ utility companies in electricity generation and distribution.

The Challenge

The client faced the complex task of ingesting data from various tenants with legacy systems into a modern data warehouse. This process needed to align with the latest data warehouse specifications efficiently and effectively while maintaining data integrity and accuracy throughout the transition.

Key Results

Reduced pipeline creation time by 40% through the implementation of the pipeline_builder library for automating the pipeline creation process.
Reduced onboarding time for a new tenant by 50% (8 weeks to 4 weeks)
Improved data accessibility and reliability for 40+ utility companies
Streamlined pipeline creation process, reducing manual coding efforts by 80%

Our Solution

To address the challenge, JashDS developed a robust tool called the pipeline_builder library. The solution involved:

Defined a standard template to capture data mapping rules
Designing an intelligent pipeline_builder library that can create data pipelines by translating data mapping rules into boilerplate pipeline code.
The automation covered 80% to 90% of standard mapping rules like renaming data columns, extracting data from a field using regex, etc. The remaining 10 to 20% of customization was the manual coding effort required by the pipeline developers.
Developing ingest, export, and master jobs to automate and streamline data processing.
Developed automated test cases from the data mapping rules. These are executed as part of the nightly integration tests and have identified several regression issues till now.

Technologies Used

GCP - Google Cloud Storage, Dataflow, Composer (Airflow), Cloud FunctionsMatillionCircleCI

Related Case Studies

← Back to All Case Studies

Data Engineering

Revolutionizing Data Infrastructure for AI-Driven Green Energy Solutions

JashDS revolutionized a green energy tech company's data infrastructure by implementing a scalable Matillion-based ETL solution and automated CI/CD processes, resulting in 2-3x faster client onboarding and a 35% reduction in Google Cloud costs. The comprehensive solution included reusable components, optimized SQL queries, and efficient data aggregation techniques, enhancing the client's ability to process vast amounts of utility data from 40+ companies and support their AI-driven green energy initiatives.

Data Engineering

Azure to AWS SaaS Platform Migration

An EdTech SaaS company migrated its entire Azure-hosted student risk monitoring platform to AWS in 9 weeks — lifting and shifting 17 VMs, 12 PostgreSQL databases, and multiple application services with zero disruption to school district operations. The solution leveraged Terraform, AWS Control Tower, and a fully automated CI/CD pipeline to deliver a scalable, cost-optimized cloud foundation built for rapid expansion.

Data Engineering

Real-Time AI Chatbot Platform’s Lambda to ECS Migration

An AI chatbot startup faced critical Lambda performance issues including 100% memory utilization causing crashes, 7-8 second cold starts,Scalability issues where in multiple concurrent users using this application concurrently faced issues to use the application which includes laginess taking too much time to get the response, application crashing  and completely non-functional WebSocket group chat due to protocol incompatibility between Socket.IO frontend and API Gateway WebSocket backend. Through a 4-week POC engagement, we successfully containerized Lambda functions to ECS Fargate, conducted systematic JMeter load testing up to 1,000 concurrent users, and delivered complete Terraform Infrastructure-as-Code, achieving 94% response time reduction (to sub-1-second), 100% cold start elimination, 0% error rate, and validated linear horizontal scalability while providing all technical documentation and architecture recommendations for production migration decision-making.

Have a similar challenge?

Connect with us