In high-risk industries like construction and manufacturing, worker safety isn’t just a priority; it’s a constant challenge. Fast-moving environments, heavy machinery, and human unpredictability make it incredibly tough to monitor compliance and catch dangerous behavior before it leads to injury.

As data scientists, we wanted to tackle that challenge head-on. What if we could use computer vision to monitor safety in real time—right at the edge?

In this post, I’ll guide you through a project where we developed a modular, reusable safety monitoring component utilizing advanced AI techniques. By combining YOLO-based SAS Event Stream Processing (ESP) on NVIDIA’s Jetson Orin edge device, we created a system capable of real-time hazard detection that can literally help save lives.

Let’s dive into how this cutting-edge blend of AI and edge computing is reshaping workplace safety—one frame at a time.

Why real-time monitoring matters

Every year, countless workplace injuries and fatalities occur. Many of these could have been avoided with better safety oversight. According to the U.S. Bureau of Labor Statistics, from 2022 to 2023, there were 3.5 fatalities per 100,000 full-time workers. Although traditional methods such as manual inspections and CCTV cameras help, they often lack the ability to provide actionable, real-time insights.

For example, a worker forgetting to wear gloves while handling dangerous equipment might go unnoticed until an incident occurs. Similarly, a worker operating machinery without a helmet could be at risk of serious head injuries from falling objects. Someone entering a hazardous zone without a high-visibility vest might go unnoticed, increasing the likelihood of accidents involving moving vehicles. Unsafe postures, such as crouching under unstable machinery or standing too close to heavy loads being lifted, are also hard to monitor continuously. These gaps in oversight highlight the critical need for automated, real-time safety monitoring.

Real-time computer vision bridges this gap by delivering instant detection and alerts. By automating safety monitoring, organizations can mitigate risks, improve compliance, and ultimately protect their workforce.

The solution: AI meets safety

1. Our solution focuses on two main capabilities:
   Personal Protective Equipment (PPE) Detection: Using You Only Look Once X (YOLOX), the system identifies whether workers are wearing required safety gear such as helmets and vests.
2. Pose Estimation: Leveraging YOLOv7 Pose, it analyzes worker movements and postures to flag potentially dangerous behaviours.

These models are deployed on SAS ESP for real-time inference, leveraging its native support for ONNXruntime. This enables us to run YOLO-based models in the Open Neural Network Exchange (ONNX) format, ensuring compatibility and efficient execution across various platforms.

Building the system

1. Model Training and Optimization

   • YOLOX for PPE Detection: YOLOX is a state-of-the-art object detection framework. To tailor it for this project, we trained it on a custom data set of workers in various industrial settings. The data set included images with and without PPE under different lighting conditions, ensuring the model performs robustly in real-world scenarios.
   • YOLOv7 Pose for Posture Monitoring: YOLOv7 Pose excels at key point detection and tracking, making it ideal for recognizing body parts, such as shoulders, elbows, hips, knees, and ankles. By applying heuristics to the model output, we can identify risks such as improper lifting techniques or workers engaging in hazardous actions.

2. Real-Time Deployment with SAS ESP

   Deploying the models through SAS ESP permitted us to use both SAS and open-source models, which enabled us to achieve:

   • Event-Driven Alerts: The system immediately notifies workers and supervisors when it detects non-compliance or unsafe behaviours.
   • Scalability: SAS ESP’s architecture supports the simultaneous processing of multiple video feeds, making it suitable for large facilities.
   • Low-Latency Inference: Processing occurs within milliseconds, ensuring alerts are timely and actionable.

3. Edge Deployment on Jetson Orin

   To ensure real-time performance and data privacy, we chose NVIDIA’s Jetson Orin. This edge-first design keeps all video and PII on-device, which is critical for regulated industries that demand auditability and compliance. A standard webcam serves as the video input, making the solution cost-effective and easy to deploy across diverse environments.

Physical setup: Bringing real-time AI to life

The physical setup for this system was designed to be both practical and scalable for industrial environments, illustrated in Figure 1. At its core is the NVIDIA Jetson Orin, a compact yet powerful edge computing device capable of running advanced AI models with low latency.

A standard webcam serves as the video input, positioned strategically to capture activity within a designated monitoring zone, where a manufacturing belt is placed, as shown in Figure 2.

A video of the demo in action, showing the workspace and some employees wearing Personal Protective Equipment (PPE) and simulating some safe and unsafe leaning behaviors with real-time alerts, can also be viewed here.

Although this setup demonstrates the concept in client deployments, we utilize IP cameras to stream footage in real-time, ensuring higher resolution, flexibility in placement, and enhanced network connectivity for robust performance in industrial environments. The area is delimited by crowd management belts. The live stream, with real-time detections and alerts, is visible on the monitor. Moreover, alerts produce an acoustic hint.

How it works: A use case

Scenario: A manufacturing plant requires workers to wear helmets and vests while operating heavy machinery.

1. Monitoring PPE Compliance
   The system continuously analyses video feeds. If it detects a worker without a vest, it sends an immediate alert to their supervisor and displays a warning on-site.
2. Tracking Unsafe Postures
   Workers' movements are tracked throughout the camera’s field of view, and bending over to lift a heavy load incorrectly triggers the pose estimation model. An alert is sent, enabling the supervisor to intervene before an injury occurs.
3. Real-Time Alerts and Reports
   All detections are logged, enabling managers to analyse trends and improve overall safety protocols.

Results and insights

The system was tested in simulated industrial settings, yielding impressive results:

• PPE Detection Accuracy: Great performance even under challenging conditions like poor lighting or partial occlusions.
• Latency: Sub-100ms inference times, ensuring near-instantaneous feedback.
• Ease of Deployment: The solution was operational within hours, requiring only minimal hardware setup and configuration.

The feedback from test users highlighted the system’s ability to proactively prevent accidents, reducing both human and financial costs.

Benefits of real-time safety monitoring

This project demonstrates how AI can fundamentally enhance workplace safety. Key advantages include:

1. Proactive Risk Mitigation: By identifying risks in real-time, the system allows for immediate corrective actions.
2. Improved Compliance: Continuous monitoring ensures adherence to safety protocols without requiring constant human supervision.
3. Scalability and Cost Efficiency: The edge-based architecture reduces infrastructure costs, making it accessible for organizations of all sizes.

Challenges and Future Directions

No system is without its challenges. For this project, the primary hurdles included:

• Data set Diversity: Capturing a sufficient number of varied examples of PPE and unsafe behaviors to ensure robust model performance was a significant challenge. To address this, we tackled data set diversity in-house by collecting a custom data set in a warehouse environment. This setup enabled us to simulate real-world conditions, including varied lighting, angles, and worker movements, ensuring the models were robust and adaptable to diverse scenarios.
• Edge Hardware Optimization: Balancing model complexity with inference speed on Jetson Orin. We fine-tuned the YOLOX-s model to maintain high accuracy while ensuring that inference times remained within acceptable limits for real-time applications.

Looking ahead, we can further augment the deployed system to:

• Integrate IoT Sensors: Combine video data with environmental sensors (such as noise levels and temperature) to create a comprehensive safety solution. This extends to PLC (Programmable Logic Controller) integration, enabling the implementation of kill switches for electronic machinery.
• Enhance Predictive Capabilities: Utilize historical data to forecast and mitigate future risks.
• Expand Deployment: Scale the system to handle multi-camera setups across larger facilities.
• Expand Use-cases: Improve the system to handle other classes of protective equipment and detect other unsafe behaviours (for example, forklift violations)

Call to Action

If you’re interested in leveraging AI to improve workplace safety, explore the following resources:

• SAS Event Stream Processing
• Worker Safety Solution from SAS and SoftServe
• Worker Safety Top Tips

Want to see the solution in action? Contact us to explore how real-time computer vision can enhance safety within your organization.

Final Thoughts

As data scientists, we have the unique privilege of transforming complex AI algorithms into practical solutions that make a tangible impact in the real world. This project showcases how cutting-edge computer vision technologies, combined with real-time processing, can proactively save lives and enhance workplace safety.

This is more than a proof of concept; it’s a foundation for scaling AI-driven safety monitoring across entire industries, standardizing compliance, and preventing injuries on a massive scale.