Enabling Real-Time Log Analytics and Security Monitoring on AWS

L&T Infrastructure Development Projects Limited (L&T IDPL), a subsidiary of Larsen & Toubro, is one of India’s leading infrastructure development organizations with projects spanning roads, bridges, airports, ports, metro rails, hydel energy, and urban infrastructure.With large-scale operations and growing digital workloads, the organization required a scalable and intelligent monitoring platform capable of delivering real-time operational visibility, proactive anomaly detection, and centralized observability across applications and infrastructure.

Business ChallengeL&T IDPL’s existing on-premises log management infrastructure lacked real-time monitoring and centralized analytics capabilities, limiting visibility into application failures, security incidents, and operational anomalies. The absence of automated monitoring required administrators to manually review log files, resulting in delayed troubleshooting, operational inefficiencies, and increased risk of missing critical events. As application and security log volumes continued to grow, the legacy infrastructure struggled with scalability, storage limitations, and delayed processing. The organization also required stronger backup, disaster recovery, encryption, and governance mechanisms to improve resilience and compliance readiness.

Goals & ObjectivesThe primary objective was to build a secure, scalable, and cloud-native log analytics platform capable of ingesting, processing, monitoring, and visualizing application and security logs in real time.The organization also aimed to improve operational visibility, strengthen security monitoring, accelerate issue resolution, and establish proactive alerting capabilities across critical systems.

Solution ApproachPentagon System & Services designed and implemented a fully AWS-native real-time log analytics architecture optimized for scalability, resilience, and operational intelligence. On-premises application and security logs were continuously streamed to AWS using Kinesis Agent and ingested into Amazon Kinesis Data Streams for real-time processing. The incoming data streams were processed using Amazon Kinesis Data Analytics for Apache Flink, deployed securely within an Amazon VPC using private subnets. The platform enabled real-time filtering, transformation, and enrichment of logs while identifying operational anomalies such as latency spikes, application failures, HTTP errors, unauthorized access attempts, and suspicious login activities Malformed or failed records were automatically redirected to Amazon SQS Dead Letter Queues (DLQ), ensuring resilience and uninterrupted processing of the primary log pipeline.

Implementation ApproachAmazon CloudWatch was implemented for centralized monitoring of infrastructure, application, and security metrics, while Amazon SNS enabled instant notifications and alerts for operational and security events. To strengthen resilience and long-term retention, processed logs and backups were securely stored in Amazon S3 with lifecycle and disaster recovery considerations. Security and governance controls were implemented using IAM policies, encryption mechanisms, access restrictions, and VPC-based network isolation to strengthen compliance and operational governance. The entire solution was designed to deliver centralized observability, real-time analytics, operational scalability, and proactive incident management.