Title: Intelligent Web Crawling and Anomaly Detection for IoT Ecosystems: A Federated Learning Approach with Explainable AI
Authors: Sachin Roy
Volume: 9
Issue: 12
Pages: 178-187
Publication Date: 2025/12/28
Abstract:
The exponential growth of the Internet of Things (IoT) and dynamic web content presents dual challenges: efficiently acquiring fresh, relevant data and securing the resultant interconnected systems. Traditional web crawlers struggle with freshness and scalability, while centralized AI models for IoT security face privacy and data silo issues. This paper proposes a novel, integrated framework that synergizes intelligent web crawling with federated learning-based anomaly detection for IoT networks. We design an "Intelligent Federated Crawler-Auditor" that employs adaptive crawlers informed by federated learning models to prioritize data sources while using a federated anomaly detection system to secure IoT edge devices generating web data. The framework integrates Explainable AI principles to make both crawling decisions and security alerts interpretable. Our implementation demonstrates a 37% improvement in content freshness metrics and a 28% increase in anomaly detection precision compared to baseline systems while maintaining data privacy through decentralized learning. This integrated approach addresses critical gaps in search engine freshness, IoT device security, and system trustworthiness, offering a blueprint for next-generation secure, intelligent web-data pipelines.