Tech Stack: Azure Data Factory, Azure Data Lakes, Azure SQL Storage, Azure Machine Learning
About the Client
Our client is a global provider of machinery in the fields of laser technology and electronics. Specific challenges around maintenance of this heavy machinery includes troubleshooting and timely repair. As a value-added service, the client offers preventative maintenance to its customers.
The client was unable to store and access data in real time to monitor machine performance. The inability to predict or prevent machine failure within SLA hours often made it difficult to provide services to customers in a timely manner.
The machinery generated a high volume of log files. The siloed and unstructured nature of the data made it difficult for the client to manage and analyze. Without the centralized analysis capabilities, they were unable to predict issues, recognize warnings, prevent incidents before they occurred, or perform remote troubleshooting.
Processing data from these log files was time-consuming and required a scalable infrastructure. We used Azure Data Lakes–storage repositories with an analytics and action purpose–as a repository alternative for this large volume of unstructured and semi-structured data. The log files were stored in Azure Blobs. From there, the data was transformed and stored in a custom dimension in Azure SQL tables.
Below is the architectural flow that demonstrates how Azure Data Factory and Azure Data Lakes were utilized to stream data collection in real time.
Here’s a glimpse of how data transformation is done in Azure Data Lake Analytics.
We used Azure Data Factory wherein the data pipeline processed 20 expression rules in real time. Ten out of 20 of these rules were implemented to detect machine problems.
We also used machine learning and predictive analytics to identify and predict errors or failures before their occurrence.
The platform enabled the client to actively monitor machinery performance remotely for faster service. The dashboards displayed color-coded warning alerts identified with data analytics on large log files generated by these machines. The predictive analysis further notified the client about machine failures and issues, enabling staff to act proactively to address problems.
The following was achieved through the implementation of our platform:
- Pro-active predicting and preventing machine failure
- Real-time analytics and performance monitoring
- Azure Data Lakes to store, analyze, and turn data into actionable insights
- Machine performance data accessibility in almost real time