Adaptive Spike Neural Networks for Natural Language Inference Tasks with Dynamic Spike Predictor

Spike Neural Networks offer energy efficiency and are promising candidates for ultra-low-power inference on neuromorphic hardware. While extensively studied in computer vision, their application in Natural Language Processing remains limited and underexplored. Three significant challenges of the existing work are as follows: (1) spike firing functions are sensitive to initial conditions, (2) spike timings are stochastic even for identical token inputs, preventing the stable preservation of contextual information, and (3) the analysis of spike occurrences on learning effectiveness is limited. To improve learning efficiency and stability, we propose Dynamic Spike Predictor (DSP) that adaptively regulates spike generation. DSP predicts a scale-adjusted input current at each time step to regulate spike activity, maintaining stable gradient flow, with only about 0.2% additional parameters to the backbone SNNs. We validate its effectiveness through comprehensive experiments on three NLI benchmarks (CB, RTE, and SICK), addressing research questions on the learning performance, robustness, and extensibility of DSP. The code is available at https://github.com/bigbases/Spike-Predictor.