Unveiling the Performance Gap Between Deep Text Mining and Human Coders in Environmental Information Extraction

Automated text mining is increasingly used in government monitoring and policy-oriented workflows, but its value depends on how closely automated labels align with human interpretation. This study compares a BiLSTM classifier with two trained human coders in classifying Korean environmental news across five categories: Climate, Air, Water, Waste/Garbage, and Others. Using a time-separated evaluation (2006–2017 training; 2018–2022 testing), the classifier achieved high accuracy (> 0.98), but lower agreement with human coders (F1 = 0.76) than coder-to-coder agreement (F1 = 0.89). Discrepancies were most evident in low-text or image-dominant articles, where human integrative reasoning captured nuanced environmental information that the classifier missed. Cofusion was highest where semantic overlap was common (e.g., Climate vs. Air), whereas Water exhibited stable agreement. These findings highlight the practical importance of human-in-the-loop validation, encompassing uncertainty-based routing and periodic agreement audits, for accountable and transparent environmental information systems. Although limited to a text-based corpus, the framework provides a foundation for future multimodal and multilingual extensions.