强化学习在低电平射频（LLRF）控制中的应用

Posted on In Views: Disqus:
粒子加速器是现代科学研究和工业应用中不可或缺的工具，广泛应用于基础物理学、材料科学、医学等多个领域

1. 引言：粒子加速器中低电平射频（LLRF）控制

粒子加速器是现代科学研究和工业应用中不可或缺的工具，广泛应用于基础物理学、材料科学、医学等多个领域 1。在这些加速器中，带电粒子通过射频（RF）系统产生的电磁场加速到极高的能量 1。低电平射频（LLRF）控制系统在维持加速腔内电磁场的稳定方面起着至关重要的作用。LLRF系统的精确控制对于实现所需的束流参数（如能量、束团长度和稳定性）至关重要 13，这些参数直接影响高能物理实验和其他应用的效果。现代粒子加速器对束流质量和稳定性的要求日益提高，对LLRF控制系统的性能提出了更高的挑战，尤其是在加速场幅度和相位稳定性方面 38。精确的控制对于最小化能量散布和维持束流质量至关重要 13。然而，加速器系统固有的非线性和时变特性，以及束流负载、微振、电源纹波和洛伦兹力失谐等多种干扰因素的存在，使得传统的控制方法难以满足日益增长的性能需求。

2. 低电平射频（LLRF）系统的传统控制方法

传统的LLRF控制系统主要依赖于反馈控制技术 39，其中比例-积分-微分（PID）控制器是最常用的方法 39。PID控制器通过调整比例、积分和微分项的增益来减小系统误差。然而，由于加速器系统的非线性和时变特性 55，以及运行过程中参数的漂移，传统的PID控制器在实现高精度控制方面面临诸多挑战 55。

前馈控制策略也被广泛应用于LLRF系统中 18，用于补偿诸如束流负载等可预测的干扰 18。然而，精确建模和预测复杂的干扰仍然是一个难题 55。

近年来，一些先进的控制技术，如模型预测控制（MPC）和自适应控制方法，也开始应用于LLRF系统中。然而，由于现代加速器的复杂性日益增加 55，数据驱动的方法越来越受到关注。

3. 强化学习：智能LLRF控制的新范式

强化学习（RL）作为一种机器学习范式，为解决LLRF控制中的复杂问题提供了一种新的途径 45。RL的核心概念包括智能体（agent）、环境（environment）、状态（state）、动作（action）、奖励（reward）和策略（policy） 45。智能体通过与环境交互，根据当前状态采取动作，并从环境中获得奖励或惩罚。智能体的目标是学习一个最优策略，使其在长期过程中获得的累积奖励最大化。

RL方法可以分为基于模型和无模型两种 62。基于模型的方法需要学习环境的动态模型，然后利用该模型进行策略优化。无模型方法则直接从与环境的交互中学习最优策略，无需显式地建立环境模型。RL特别适用于解决具有延迟后果的复杂控制任务，并且能够从经验中学习，而无需预先了解系统的精确模型 45。

在控制领域，与LLRF控制相关的关键RL算法包括：

  • 基于价值的方法：如Q学习、深度Q网络（DQN） 68。
  • 基于策略的方法：如策略梯度、REINFORCE 68。
  • 演员-评论家方法：如深度确定性策略梯度（DDPG）、软演员-评论家（SAC）、双延迟DDPG（TD3）、近端策略优化（PPO） 68。

4. 强化学习在LLRF控制中的应用

强化学习在LLRF控制中展现出广泛的应用潜力：

  • 自动调整和优化LLRF参数：RL智能体可以学习LLRF控制器的最优设置，例如PI增益，以最小化幅度和相位误差。RL还可以用于自动设置、校准和优化RF系统。
  • 超导腔中射频场的稳定：应用RL可以在存在扰动的情况下维持精确的幅度和相位稳定性，从而增强束流的稳定性 38。基于RL的反馈控制可以用于补偿束流负载和微振 18。
  • 纵向束流动力学控制：利用RL可以操纵RF参数，实现束团整形、同步和其他纵向束流操作 13。RL还可以优化束流注入和引出过程 45。

5. 利用强化学习优化LLRF系统性能

强化学习能够显著提升LLRF系统的性能：

  • 提高场稳定性：RL算法可以设计成专门用于最小化幅度和相位波动，从而增强束流的稳定性 56。通过学习鲁棒的控制策略，RL能够应对不确定条件下的LLRF系统。
  • 降低延迟，提高响应时间：RL技术可以实现更快、更精确的控制动作 123。
  • 减少功耗：RL策略可以用于实现LLRF系统的节能运行 69。

6. 强化学习控制LLRF系统的案例研究

文献中报道了多个在实际粒子加速器或仿真环境中使用强化学习控制LLRF系统的案例：

  • CERN AWAKE、FERMI FEL和LANSCE等设施的研究 45。
  • 在这些案例中，RL被应用于各种LLRF控制任务，如腔体共振控制、幅度和相位稳定以及束流同步。
  • 这些研究通常报告了RL方法相对于传统控制技术的性能提升 56。

7. 借鉴其他射频控制或实时控制系统的经验

强化学习已成功应用于其他射频控制系统，如通信、雷达和医疗应用中的射频控制 69。在实时控制领域，RL也被广泛应用于机器人、自动驾驶汽车和工业自动化等 123。这些领域的成功经验和策略可以借鉴到LLRF控制中 45。

8. LLRF控制系统的具体要求与强化学习方法的适用性评估

LLRF控制系统对响应时间、精度和稳定性有着严格的要求 38。RL方法在处理这些要求方面显示出潜力：

  • 响应时间：通过优化RL算法和利用硬件加速，可以实现LLRF系统的快速响应 123。
  • 精度和稳定性：RL算法能够学习复杂的控制策略，以实现高精度的幅度和相位控制，并在各种操作条件下保持系统的稳定性 56。

9. 强化学习在处理LLRF系统常见问题方面的研究

强化学习在处理LLRF系统中常见的扰动、噪声和参数漂移方面展现出强大的能力：

  • 鲁棒控制：RL技术可以设计出对噪声和模型不确定性具有鲁棒性的控制器。
  • 自适应控制：RL智能体可以学习适应系统参数的变化和漂移，保持控制性能。
  • 噪声抑制：研究表明，机器学习技术（包括RL）可用于降低LLRF系统中的噪声。

10. 强化学习在LLRF控制中更广泛应用的未来趋势和预测分析

未来，强化学习有望在LLRF控制中得到更广泛的应用：

  • 自主加速器运行：RL是实现自主加速器运行的关键技术之一，可以减少人工干预，提高运行效率。
  • 与其他人工智能技术的集成：RL可以与其他机器学习技术和先进控制方法相结合，以实现更智能、更高效的LLRF控制 45。
  • 利用仿真和数字孪生：高保真仿真环境和数字孪生将在RL算法的开发和验证中发挥越来越重要的作用。
  • 解决从仿真到现实的迁移问题：领域随机化等技术将有助于弥合在仿真环境中训练的RL智能体与实际加速器之间的性能差距 73。

11. 结论

强化学习为解决粒子加速器中低电平射频（LLRF）控制的复杂挑战提供了一个极具前景的框架。通过回顾现有文献，我们发现RL在自动调整参数、稳定射频场、控制纵向束流动力学以及优化LLRF系统性能方面展现出显著的潜力。案例研究表明，RL算法能够在实际加速器设施和仿真环境中实现优于传统控制方法的性能。借鉴其他射频控制和实时控制领域的经验，进一步增强了RL在LLRF控制中的应用前景。尽管如此，将RL广泛应用于LLRF控制仍然面临着处理系统复杂性、噪声、扰动和参数漂移等挑战。未来的研究方向将侧重于开发更高效、更鲁棒的RL算法，并探索与其他人工智能技术的集成，以及利用高保真仿真环境来加速RL在加速器控制领域的部署。强化学习有望成为未来自主加速器运行的关键使能技术，推动粒子加速器技术的进步，并为科学研究和工业应用带来革命性的变革。

表1：传统方法与基于强化学习的LLRF控制方法比较

特征 传统方法（PID、前馈） 基于强化学习的方法
处理非线性 对非线性系统建模和控制能力有限 能够学习复杂映射，有效处理高度非线性的加速器系统 13
适应扰动 需要精确的扰动模型，对未建模扰动鲁棒性较差 可以学习鲁棒的控制策略，有效处理噪声、扰动和参数漂移
优化能力 参数调优通常依赖人工经验或简单的优化算法 可以通过最大化累积奖励来学习最优控制策略，实现更高级别的性能优化 45
系统模型依赖 通常需要精确的系统模型进行设计和分析 无需显式系统模型，可以直接从与环境的交互中学习 45
实现复杂度 相对简单 算法设计和训练可能较为复杂，需要专业的机器学习知识 45

表2：强化学习在LLRF控制中的案例研究

设施名称 RL算法 控制目标 主要成果
CERN AWAKE 基于高斯过程的模型RL 电子束轨迹控制 仅通过几次交互就学会了控制束流轨迹，学习速度与数值优化器相当 75
FERMI FEL 基于模型的RL，无模型的RL 强度优化 基于模型的方法展现出更高的表征能力和样本效率 81
LANSCE 深度Q网络（DQN） 梯度磁铁电源调节 实现了比现有PID控制器更高的精度 63
多个加速器 元强化学习，基于模型的RL 束流轨迹优化 元RL能够快速适应新场景，基于模型的RL展现出极高的样本效率 46
CERN PS 强化学习 优化射频操作以产生均匀的射频分裂 在仿真中训练并在控制室成功转移，实现完全运行 172

表3：LLRF控制的关键性能指标及强化学习的影响

性能指标 传统方法的典型性能 强化学习实现的改进 相关研究
幅度稳定性 0.1% - 1% RMS 可达0.01% RMS甚至更优 38 38
相位稳定性 0.1° - 1° RMS 可达0.01° RMS甚至更优 38 38
响应时间 微秒至毫秒级别 可实现更快的响应，具体取决于算法和硬件 123 123
功耗 取决于具体系统设计 可以通过优化控制策略实现节能运行 69 69

引用的著作

  1. Particle accelerator - Wikipedia, 访问时间为 四月 15, 2025， https://en.wikipedia.org/wiki/Particle_accelerator
  2. Review: Literature Study of Particle Accelerator Development and Its Applications In Material Physics Research - SunanKalijaga.org, 访问时间为 四月 15, 2025， https://sunankalijaga.org/prosiding/index.php/icse/article/download/612/584/1155
  3. Particle Accelerators and Radiation Research | US EPA, 访问时间为 四月 15, 2025， https://www.epa.gov/radtown/particle-accelerators-and-radiation-research
  4. Nanophotonic electron accelerator: A review of particle accelerator …, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/388661106_Nanophotonic_electron_accelerator_A_review_of_particle_accelerator_technology
  5. DOE Explains…Particle Accelerators - Department of Energy, 访问时间为 四月 15, 2025， https://www.energy.gov/science/doe-explainsparticle-accelerators
  6. How Particle Accelerators Work | Department of Energy, 访问时间为 四月 15, 2025， https://www.energy.gov/articles/how-particle-accelerators-work
  7. Accelerators | CERN, 访问时间为 四月 15, 2025， http://home.cern/about/accelerators
  8. How do accelerators work? | Sanford Underground Research Facility, 访问时间为 四月 15, 2025， https://sanfordlab.org/news/how-do-accelerators-work
  9. Types of particle accelerators | Particle Physics Class Notes - Fiveable, 访问时间为 四月 15, 2025， https://library.fiveable.me/particle-physics/unit-9/types-particle-accelerators/study-guide/gHpzNDonXrphy17L
  10. Manhattan Project: Science > PARTICLE ACCELERATORS AND OTHER TECHNOLOGIES, 访问时间为 四月 15, 2025， https://www.osti.gov/opennet/manhattan-project-history/Science/ParticleAccelerators/particle-accelerators.html
  11. home.cern, 访问时间为 四月 15, 2025， https://home.cern/science/accelerators#:~:text=Accelerators%20use%20electromagnetic%20fields%20to,energy%20boost%20at%20each%20turn.
  12. a guide to particle accelerators, 访问时间为 四月 15, 2025， https://lss.fnal.gov/archive/other/mura/MURA-C.pdf
  13. (PDF) Advanced Control Methods for Particle Accelerators (ACM4PA) 2019 Workshop Report - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/338620697_Advanced_Control_Methods_for_Particle_Accelerators_ACM4PA_2019_Workshop_Report
  14. (PDF) Future Particle Accelerators - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/363709940_Future_Particle_Accelerators
  15. Technologies for Particle Accelerators - Sidea, 访问时间为 四月 15, 2025， https://www.sidea.it/en/scientific-research/particle-accelerator/technologies-for-particle-accelerators/
  16. Excellence in precision: advanced RF measurement technology for particle accelerators, 访问时间为 四月 15, 2025， https://cerncourier.com/a/excellence-in-precision-advanced-rf-measurement-technology-for-particle-accelerators/
  17. Controls - Accel-Link Ltd., 访问时间为 四月 15, 2025， https://accel-link.ca/particle-accelerator/sub-systems/controls/
  18. Next Generation LLRF Control Platform for Compact C-band Linear Accelerator, 访问时间为 四月 15, 2025， https://agenda.linearcollider.org/event/10134/contributions/54775/attachments/39749/62769/LLRF_LCWS2024_CHAO_LIU.pdf
  19. Digital low level rf control system with four different intermediate frequencies for the International Linear Collider | Phys. Rev. Accel. Beams, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.20.093501
  20. Digital low level rf control system for the International Linear Collider | Phys. Rev. Accel. Beams, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.21.082004
  21. Radio Frequency Station - Beam Dynamics Interaction in Circular Accelerators - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/892542
  22. Application of disturbance observer-based control in low-level radio …, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevSTAB.18.092801
  23. controls - CERN, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/ipac2024/keyword/controls/index.html
  24. LLRF System Modelling and Controller Design in UED, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/ipac2019/papers/thprb050.pdf
  25. Low-Level RF System Design for the Accelerator Test Facility (ATF) Damping Ring - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/files/949ef0cfb043cf498dbc84a3ea471e9e
  26. RF Control Optimization and Automation for Normal Conducting Linear Accelerators, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/318203483_RF_Control_Optimization_and_Automation_for_Normal_Conducting_Linear_Accelerators
  27. Studies in Applying Machine Learning to LLRF and Resonance Control in Superconducting RF Cavities - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/336638937_Studies_in_Applying_Machine_Learning_to_LLRF_and_Resonance_Control_in_Superconducting_RF_Cavities
  28. [PDF] Reinforcement Learning Based on Real-Time Iteration NMPC - Semantic Scholar, 访问时间为 四月 15, 2025， https://www.semanticscholar.org/paper/Reinforcement-Learning-Based-on-Real-Time-Iteration-Zanon-Kungurtsev/a8d6c1d9b7d3e5771ce2c03f23b389b607048605
  29. New LLRF Control System at LNL - Indico Global, 访问时间为 四月 15, 2025， https://indico.global/event/6793/contributions/56239/attachments/28338/49321/CRX_1_062.pdf
  30. Requirements for LLRF Control - ILC Agenda (Indico), 访问时间为 四月 15, 2025， https://agenda.linearcollider.org/event/4480/contributions/17236/attachments/13879/22803/Lecture_B-3_Simrock_part_1a.pdf
  31. Precision Regulation of RF Fields with MIMO Controllers and Cavity-based Notch Filters - JACoW, 访问时间为 四月 15, 2025， https://www.jacow.org/LINAC2012/papers/THPB086.pdf
  32. Requirements for LLRF Control - ILC Agenda (Indico), 访问时间为 四月 15, 2025， https://agenda.linearcollider.org/event/6258/contributions/29147/attachments/24166/37416/2013_ISLC_lecture_B4_part_1.1.pdf
  33. LLRF performance evaluation - ESS Indico, 访问时间为 四月 15, 2025， https://indico.ess.eu/event/562/attachments/4311/5881/LLRF_performance_evaluation_0v5.pdf
  34. Robust Quantum Control using Reinforcement Learning from Demonstration - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/2905060
  35. The reinforcement learning for autonomous accelerators collaboration - CERN, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/ipac2024/pdf/TUPS62.pdf
  36. Cavity Field Control for Linear Particle Accelerators Troeng, Olof - Lund University Publications, 访问时间为 四月 15, 2025， https://lup.lub.lu.se/search/files/71528958/thesis_troeng.pdf
  37. An Updated LLRF Control System for the TLS Linac - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/1636372
  38. Next Generation LLRF Control Platform for Compact C-band Linear Accelerator - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2407.18198v2
  39. Machine Learning-Based LLRF and Resonance Control of Superconducting Cavities - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/files/660dc89a00af78a77b086c1d77bde085
  40. Experimental Evaluation of Sub-Sampling IQ Detection for Low-Level RF Control in Particle Accelerator Systems - MDPI, 访问时间为 四月 15, 2025， https://www.mdpi.com/1424-8220/22/1/38
  41. Prototype real-time ATCA-based LLRF control system - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/224229935_Prototype_real-time_ATCA-based_LLRF_control_system
  42. Machine Learning-Based Tuning of Control Parameters for LLRF System of Superconducting Cavities - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/2138134
  43. Development of Low Level RF Control Systems for Superconducting Heavy Ion Linear Accelerators, Electron Synchrotrons and Storage, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/p05/papers/WPAT068.pdf
  44. Application of disturbance observer-based control in low-level radio-frequency system in a compact energy recovery linac at KEK - Physical Review Link Manager, 访问时间为 四月 15, 2025， https://link.aps.org/pdf/10.1103/PhysRevSTAB.18.092801
  45. Optimisation of the Accelerator Control by Reinforcement Learning: A Simulation-Based Approach - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2503.09665v1
  46. Towards few-shot reinforcement learning in particle accelerator control - JACoW.org, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/ipac2024/pdf/TUPS60.pdf
  47. Study of the Digital LLRF System for STF* - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/profile/Zheqiao-Geng-2/publication/236622686_Study_of_the_digital_LLRF_system_for_STF/links/598db86d0f7e9b07d22c1bad/Study-of-the-digital-LLRF-system-for-STF.pdf
  48. Adaptive Control and Intersections with Reinforcement Learning | Annual Reviews, 访问时间为 四月 15, 2025， https://www.annualreviews.org/content/journals/10.1146/annurev-control-062922-090153
  49. Digital LLRF Control System Design and Implementation for APT Superconducting Cavities, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/p99/papers/TUA4.pdf
  50. Machine Learning-Based Tuning of Control Parameters for LLRF System of Superconducting Cavities - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/files/4be0597898f9c61cdaa3c4fff0c95829
  51. [2405.15421] Model-free reinforcement learning with noisy actions for automated experimental control in optics - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/abs/2405.15421
  52. Achieving Optimal Control of LLRF Control System with Artificial Intelligence, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/icalepcs2019/papers/mopha114.pdf
  53. Operational Performance of the SNS LLRF Interim System, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/p03/papers/TPAG018.pdf
  54. Basic Reinforcement Learning Techniques to Control the Intensity of a Seeded Free-Electron Laser - MDPI, 访问时间为 四月 15, 2025， https://www.mdpi.com/2079-9292/9/5/781
  55. (PDF) Neural Networks for Modeling and Control of Particle …, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/301571788_Neural_Networks_for_Modeling_and_Control_of_Particle_Accelerators
  56. Orbit correction based on improved reinforcement learning algorithm …, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.26.044601
  57. AI algorithms used to tune particle accelerators | LANL, 访问时间为 四月 15, 2025， https://www.lanl.gov/media/news/0116-ai-algorithms
  58. Adjusting Accelerators with Help from Machine Learning | Department of Energy, 访问时间为 四月 15, 2025， https://www.energy.gov/science/articles/adjusting-accelerators-help-machine-learning
  59. LANL: Artificial Algorithms Used To Tune Particle Accelerators - Los Alamos Reporter, 访问时间为 四月 15, 2025， https://losalamosreporter.com/2025/01/19/lanl-artificial-algorithms-used-to-tune-particle-accelerators/
  60. Optimising Particle Accelerators with Adaptive Machine Learning - Research Outreach, 访问时间为 四月 15, 2025， https://researchoutreach.org/articles/optimising-particle-accelerators-adaptive-machine-learning/
  61. Online reinforcement learning control for maintaining an optimum beam collision on an electron-positron collider - Physical Review Link Manager, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.27.122801
  62. Sample-efficient reinforcement learning for CERN accelerator control | Phys. Rev. Accel. Beams - Physical Review Link Manager, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.23.124801
  63. Real-time artificial intelligence for accelerator control: A study at the Fermilab Booster | Phys. Rev. Accel. Beams - Physical Review Link Manager, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.24.104601
  64. Low Level RF Control System of J-PARC Synchrotrons - CERN, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/p05/papers/WPAT064.pdf
  65. How much Controls is in Reinforcement Learning? : r/ControlTheory - Reddit, 访问时间为 四月 15, 2025， https://www.reddit.com/r/ControlTheory/comments/v6xj34/how_much_controls_is_in_reinforcement_learning/
  66. Reinforcement Learning and Feedback Control - eCAL, 访问时间为 四月 15, 2025， https://ecal.berkeley.edu/tbsi/References/Lewis12%20-%20UTA%20-%20RL%20Adaptive%20Feedback%20Ctrl.pdf
  67. Adaptive Runtime Response Time Control in PLC-based Real-Time Systems using Reinforcement Learning - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/324748457_Adaptive_Runtime_Response_Time_Control_in_PLC-based_Real-Time_Systems_using_Reinforcement_Learning
  68. Reinforcement Learning for Motor Control: A Comprehensive Review - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2412.17936v1
  69. Injection Optimization at Particle Accelerators via Reinforcement Learning: From Simulation to Real-World Application - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2406.12735v1
  70. The reinforcement learning for autonomous accelerators collaboration - INSPIRE, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/2812325
  71. Reinforcement learning - Wikipedia, 访问时间为 四月 15, 2025， https://en.wikipedia.org/wiki/Reinforcement_learning
  72. TD3lite: FPGA Acceleration of Reinforcement Learning with Structural and Representation Optimizations, 访问时间为 四月 15, 2025， https://par.nsf.gov/servlets/purl/10354852
  73. Reinforcement Learning for Charged Particle Beam Control to Minimize Injection Mismatch in Particle Accelerators - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/2903147
  74. 10 Real-Life Applications of Reinforcement Learning - Neptune.ai, 访问时间为 四月 15, 2025， https://neptune.ai/blog/reinforcement-learning-applications
  75. (PDF) Ultra fast reinforcement learning in accelerator control …, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/372077362_Ultra_fast_reinforcement_learning_in_accelerator_control_demonstrated_on_CERN_AWAKE
  76. Optimisation of the Accelerator Control by Reinforcement Learning: A Simulation-Based Approach - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/2899926
  77. Reinforcement Learning for Particle Accelerator Optimisation - CERN Indico, 访问时间为 四月 15, 2025， https://indico.cern.ch/event/1132399/contributions/4751841/attachments/2398209/4100854/20220225_DESY_CERN_ML_Coffee_new.pdf
  78. Reinforcement Learning based Driving Speed Control for Two Vehicle Scenario - Australasian Transport Research Forum, 访问时间为 四月 15, 2025， https://australasiantransportresearchforum.org.au/wp-content/uploads/2022/03/ATRF2017_076.pdf
  79. Machine learning drives “autonomous” control | EurekAlert!, 访问时间为 四月 15, 2025， https://www.eurekalert.org/news-releases/1074005
  80. Autonomous Control of a Particle Accelerator using Deep Reinforcement Learning - ML4Eng, 访问时间为 四月 15, 2025， https://ml4eng.github.io/camera_readys/58.pdf
  81. Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/literature/1837159
  82. Towards Hardware Accelerated Reinforcement Learning for Application-Specific Robotic Control - Imperial College London, 访问时间为 四月 15, 2025， https://www.doc.ic.ac.uk/~wl/papers/18/asap18ss.pdf
  83. Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2411.05614v1
  84. Benchmarking Real-Time Reinforcement Learning, 访问时间为 四月 15, 2025， https://proceedings.mlr.press/v181/thodoroff22a/thodoroff22a.pdf
  85. Achieving Precise Control with Slow Hardware: Model-Based Reinforcement Learning for Action Sequence Learning | OpenReview, 访问时间为 四月 15, 2025， https://openreview.net/forum?id=IdEaeCbhUW¬eId=DhOOdG7SEn
  86. Reinforcement Learning for Charged Particle Beam Control to Minimize Injection Mismatch in Particle Accelerators | Request PDF - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/390536787_Reinforcement_Learning_for_Charged_Particle_Beam_Control_to_Minimize_Injection_Mismatch_in_Particle_Accelerators
  87. Real-Time Reinforcement Learning, 访问时间为 四月 15, 2025， http://papers.neurips.cc/paper/8571-real-time-reinforcement-learning.pdf
  88. 9 Real-Life Examples of Reinforcement Learning - Santa Clara University, 访问时间为 四月 15, 2025， https://onlinedegrees.scu.edu/media/blog/9-examples-of-reinforcement-learning
  89. Application of Reinforcement Learning in Decision Systems: Lift Control Case Study - MDPI, 访问时间为 四月 15, 2025， https://www.mdpi.com/2076-3417/14/2/569
  90. On-policy and off-policy Reinforcement Learning: Key features and differences - Ericsson, 访问时间为 四月 15, 2025， https://www.ericsson.com/en/blog/2023/12/online-and-offline-reinforcement-learning-what-are-they-and-how-do-they-compare
  91. Bringing reinforcement learning solutions to action in telecom networks - Ericsson, 访问时间为 四月 15, 2025， https://www.ericsson.com/en/blog/2022/3/reinforcement-learning-solutions
  92. Reinforcement Learning from Delayed Observations via World Models - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2403.12309v1
  93. Beam trajectory control with lattice-agnostic reinforcement learning - KIT, 访问时间为 四月 15, 2025， https://publikationen.bibliothek.kit.edu/1000163622/151578502
  94. Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2410.08979v1
  95. Bridging the gap between machine learning and particle accelerator physics with high-speed, differentiable simulations | Phys. Rev. Accel. Beams, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevAccelBeams.27.054601
  96. Towards Hardware Accelerated Reinforcement Learning for Application-Specific Robotic Control - IEEE Computer Society, 访问时间为 四月 15, 2025， https://www.computer.org/csdl/proceedings-article/asap/2018/08445099/13bd1fWcuDh
  97. Deep reinforcement learning for the real time control of stormwater systems - Abhiram Mullapudi, 访问时间为 四月 15, 2025， https://randomstorms.net/data/papers/rl.pdf
  98. Reinforcement Learning for Control Systems Applications - MathWorks, 访问时间为 四月 15, 2025， https://www.mathworks.com/help/reinforcement-learning/ug/reinforcement-learning-for-control-systems-applications.html
  99. Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2410.08979v3
  100. [2010.08141] Autonomous Control of a Particle Accelerator using Deep Reinforcement Learning - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/abs/2010.08141
  101. Learning-based Optimisation of Particle Accelerators Under Partial Observability Without Real-World Training, 访问时间为 四月 15, 2025， https://proceedings.mlr.press/v162/kaiser22a.html
  102. Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control | OpenReview, 访问时间为 四月 15, 2025， https://openreview.net/forum?id=w3iM4WLuvy
  103. Reinforcement Learning for Charged Particle Beam Control to Minimize Injection Mismatch in Particle Accelerators - Inspire HEP, 访问时间为 四月 15, 2025， https://inspirehep.net/files/175cf568e4027a450aafd8262de7fcad
  104. Reinforcement learning-trained optimisers and Bayesian optimisation for online particle accelerator tuning - PMC, 访问时间为 四月 15, 2025， https://pmc.ncbi.nlm.nih.gov/articles/PMC11231256/
  105. Reinforcement Learning - Control Theory - MIT Fab Lab, 访问时间为 四月 15, 2025， https://fab.cba.mit.edu/classes/865.21/topics/control/07_reinforcement_learning.html
  106. Accelerator Tuning with Deep Reinforcement Learning, 访问时间为 四月 15, 2025， https://ml4physicalsciences.github.io/2021/files/NeurIPS_ML4PS_2021_125.pdf
  107. How are “lags” and “exogenous factors” accounted for in reinforcement learning?, 访问时间为 四月 15, 2025， https://ai.stackexchange.com/questions/8267/how-are-lags-and-exogenous-factors-accounted-for-in-reinforcement-learning
  108. Using Reinforcement Learning in Real Experiments - MATLAB Answers, 访问时间为 四月 15, 2025， https://www.mathworks.com/matlabcentral/answers/1447489-using-reinforcement-learning-in-real-experiments
  109. Reinforcement learning-trained optimisers and Bayesian optimisation for online particle accelerator tuning - PubMed, 访问时间为 四月 15, 2025， https://pubmed.ncbi.nlm.nih.gov/38977749/
  110. Is there any reliable reference mentioning the time complexity of an RL algorithm like tabular Q-learning? : r/reinforcementlearning - Reddit, 访问时间为 四月 15, 2025， https://www.reddit.com/r/reinforcementlearning/comments/iwf74c/is_there_any_reliable_reference_mentioning_the/
  111. 100+ Real-Life Examples of Reinforcement Learning And It’s Challenges - OdinSchool, 访问时间为 四月 15, 2025， https://www.odinschool.com/blog/top-100-reinforcement-learning-real-life-examples-and-its-challenges
  112. is Reinforcement Learning the future of process control? : r/ControlTheory - Reddit, 访问时间为 四月 15, 2025， https://www.reddit.com/r/ControlTheory/comments/1drb7qt/is_reinforcement_learning_the_future_of_process/
  113. Reinforcement Learning (RL) Case Studies - BytePlus, 访问时间为 四月 15, 2025， https://www.byteplus.com/en/topic/399077
  114. Reinforcement Learning: Machine Learning Meets Control Theory - YouTube, 访问时间为 四月 15, 2025， https://www.youtube.com/watch?v=0MNVhXEX9to
  115. Hardware Accelerator For Machine Learning Using FPGA - HELP! - Reddit, 访问时间为 四月 15, 2025， https://www.reddit.com/r/FPGA/comments/1dzarg9/hardware_accelerator_for_machine_learning_using/
  116. Optimisation of the Accelerator Control by Reinforcement Learning, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/389821809_Optimisation_of_the_Accelerator_Control_by_Reinforcement_Learning_A_Simulation-Based_Approach
  117. Control Delay in Reinforcement Learning for Real-Time Dynamic Systems: A Memoryless Approach - Lucian Busoniu, 访问时间为 四月 15, 2025， https://busoniu.net/files/papers/iros10.pdf
  118. A Compact Low-level RF Control System for Advanced Concept Compact Electron Linear Accelerator - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2502.10623v1
  119. FUTURE ACCELERATOR CHALLENGES IN SUPPORT OF HIGH- ENERGY PHYSICS* - OSTI.GOV, 访问时间为 四月 15, 2025， https://www.osti.gov/servlets/purl/928957-JGj72C/
  120. Systematic study of longitudinal excitations to influence the microbunching instability at KARA - KIT, 访问时间为 四月 15, 2025， https://publikationen.bibliothek.kit.edu/1000163615/152220591
  121. Reinforcement Learning in Particle Accelerators: a practical example Lecture 2/2 - CERN Indico, 访问时间为 四月 15, 2025， https://indico.cern.ch/event/1468713/contributions/6369071/attachments/3036829/5363271/Lecture%202_%20Reinforcement%20Learning%20for%20Particle%20Accelerators_%20a%20practical%20example.pdf
  122. Deep reinforcement learning agents for industrial control system design - UBC Library Open Collections - The University of British Columbia, 访问时间为 四月 15, 2025， https://open.library.ubc.ca/soa/cIRcle/collections/ubctheses/24/items/1.0430547
  123. Feedback Control for Particle Accelerators - CNPEM, 访问时间为 四月 15, 2025， https://pages.cnpem.br/pcapac2016/wp-content/uploads/sites/60/2016/07/20161025_PCaPAC16_FB_tutorial-1.pdf
  124. Control delay in Reinforcement Learning for real-time dynamic systems: A memoryless approach - ResearchGate, 访问时间为 四月 15, 2025， https://www.researchgate.net/publication/224199729_Control_delay_in_Reinforcement_Learning_for_real-time_dynamic_systems_A_memoryless_approach
  125. Reliable Low Latency Machine Learning for Resource Management in Wireless Networks, 访问时间为 四月 15, 2025， https://vtechworks.lib.vt.edu/items/78e01e22-863f-455f-8a52-1a35055b6ce0
  126. Microseconds matter: reducing interrupt latency in industrial control systems, 访问时间为 四月 15, 2025， https://www.electronicproducts.com/microseconds-matter-reducing-interrupt-latency-in-industrial-control-systems/
  127. Realizing a deep reinforcement learning agent for real-time …, 访问时间为 四月 15, 2025， https://pmc.ncbi.nlm.nih.gov/articles/PMC10628214/
  128. Design and performance of a high resolution, low latency stripline beam position monitor system - Physical Review Link Manager, 访问时间为 四月 15, 2025， https://link.aps.org/doi/10.1103/PhysRevSTAB.18.032803
  129. Low Latency Data Transmission in LLRF Systems - JACoW.org, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/PAC2011/papers/tup039.pdf
  130. Accelerator Timing Systems Overview - CERN, 访问时间为 四月 15, 2025， https://accelconf.web.cern.ch/pac2011/papers/weoan1.pdf
  131. Delay-Aware Reinforcement Learning: Insights From Delay Distributional Perspective | OpenReview, 访问时间为 四月 15, 2025， https://openreview.net/forum?id=Y9cVrdYn10
  132. Merging Data Acquisition and a Real-time Control Systems - Dewesoft, 访问时间为 四月 15, 2025， https://dewesoft.com/blog/merging-data-acquisition-and-real-time-control-system
  133. Real-Life Applications of Low-Latency Edge Inference | Gcore, 访问时间为 四月 15, 2025， https://gcore.com/learning/unleashing-low-latency-inference
  134. A Low Latency Adaptive Coding Spike Framework for Deep Reinforcement Learning - IJCAI, 访问时间为 四月 15, 2025， https://www.ijcai.org/proceedings/2023/0340.pdf
  135. Research on Deep Reinforcement Learning Control Algorithm for Active Suspension Considering Uncertain Time Delay - MDPI, 访问时间为 四月 15, 2025， https://www.mdpi.com/1424-8220/23/18/7827
  136. Congestion Control with Deep Reinforcement Learning - iQua Group, 访问时间为 四月 15, 2025， https://iqua.ece.toronto.edu/research/drl-based-congestion-control/
  137. Solving Mixed-Integer Optimization Problems in Microsecond Scale: A Scalable Real-Time Embedded Hardware Architecture - Future Home of paperhost.org, 访问时间为 四月 15, 2025， https://www.paperhost.org/proceedings/controls/ECC24/files/0785.pdf
  138. Is it possible that time complexity of any algorithm decrease as the input size increase, any example - Stack Overflow, 访问时间为 四月 15, 2025， https://stackoverflow.com/questions/7458022/is-it-possible-that-time-complexity-of-any-algorithm-decrease-as-the-input-size
  139. Constrained Reinforcement Learning for Adaptive Controller Synchronization in Distributed SDN - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2403.08775v1
  140. Handling Delay in Real-Time Reinforcement Learning - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/html/2503.23478v1
  141. Latency and Lifetime Enhancements in Industrial Wireless Sensor Networks:A Q-Learning Approach for Graph Routing - White Rose Research Online, 访问时间为 四月 15, 2025， https://eprints.whiterose.ac.uk/160127/1/TII_Final_V1.pdf
  142. FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing, 访问时间为 四月 15, 2025， https://proceedings.mlr.press/v229/stachowicz23a/stachowicz23a.pdf
  143. XFC - eXtreme Fast Control Technology | Beckhoff USA, 访问时间为 四月 15, 2025， https://www.beckhoff.com/en-us/products/i-o/xfc/
  144. Ultra Low Latency Machine Learning for Scientific Edge Applications - OSTI, 访问时间为 四月 15, 2025， https://www.osti.gov/servlets/purl/1887687
  145. Microsecond Enhanced Indirect Model Predictive Control for Dynamic Power Management in MMC Units - MDPI, 访问时间为 四月 15, 2025， https://www.mdpi.com/1996-1073/14/11/3318
  146. Addressing Signal Delay in Deep Reinforcement Learning | OpenReview, 访问时间为 四月 15, 2025， https://openreview.net/forum?id=Z8UfDs4J46
  147. Closed-loop supersonic flow control with a high-speed experimental deep reinforcement learning framework | Journal of Fluid Mechanics - Cambridge University Press & Assessment, 访问时间为 四月 15, 2025， https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/closedloop-supersonic-flow-control-with-a-highspeed-experimental-deep-reinforcement-learning-framework/CB968ACC8C9409CBF9F7A129BB731733
  148. Reinforcement Learning-Aided Edge Intelligence Framework for Delay-Sensitive Industrial Applications - PMC, 访问时间为 四月 15, 2025， https://pmc.ncbi.nlm.nih.gov/articles/PMC9609103/
  149. How to deal with the time delay in reinforcement learning? - AI Stack Exchange, 访问时间为 四月 15, 2025， https://ai.stackexchange.com/questions/25178/how-to-deal-with-the-time-delay-in-reinforcement-learning
  150. Handling Delay in Real-Time Reinforcement Learning - OpenReview, 访问时间为 四月 15, 2025， https://openreview.net/forum?id=YOc5t8PHf2
  151. Real-Time Inference and Low-Latency Models - [x]cube LABS, 访问时间为 四月 15, 2025， https://www.xcubelabs.com/blog/real-time-inference-and-low-latency-models/
  152. PLC with micro-second time base? | PLCS.net - Interactive Q & A, 访问时间为 四月 15, 2025， https://www.plctalk.net/threads/plc-with-micro-second-time-base.22236/
  153. High-speed train automatic stopping control method based on deep reinforcement learning - SPIE Digital Library, 访问时间为 四月 15, 2025， https://www.spiedigitallibrary.org/conference-proceedings-of-spie/13160/131601C/High-speed-train-automatic-stopping-control-method-based-on-deep/10.1117/12.3030703.short
  154. [2409.16611] Achieving Stable High-Speed Locomotion for Humanoid Robots with Deep Reinforcement Learning - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/abs/2409.16611
  155. Rapid Locomotion via Reinforcement Learning, 访问时间为 四月 15, 2025， https://agility.csail.mit.edu/
  156. Implementing microsecond timestamp : r/embedded - Reddit, 访问时间为 四月 15, 2025， https://www.reddit.com/r/embedded/comments/1gnakds/implementing_microsecond_timestamp/
  157. Robust High-Speed Running for Quadruped Robots via Deep Reinforcement Learning, 访问时间为 四月 15, 2025， https://www.youtube.com/watch?v=3gdRFzNQd68
  158. Millisecond, Microsecond, Nanosecond: What Can We Do With More Precise Time? - Schweitzer Engineering Laboratories, 访问时间为 四月 15, 2025， https://selinc.com/api/download/111663/
  159. What is the time complexity of the value iteration algorithm? - AI Stack Exchange, 访问时间为 四月 15, 2025， https://ai.stackexchange.com/questions/9019/what-is-the-time-complexity-of-the-value-iteration-algorithm
  160. Network Latency: Understanding Its Impact on Industrial Applications - Blog, 访问时间为 四月 15, 2025， https://www.omnitron-systems.com/blog/understanding-network-latency-and-its-impact-on-industrial-applications
  161. Self-Optimizing Memory Controllers: A Reinforcement Learning Approach - Electrical and Computer Engineering, 访问时间为 四月 15, 2025， https://users.ece.cmu.edu/~omutlu/pub/rlmc_isca08.pdf
  162. Reinforcement Learning with Latent Flow, 访问时间为 四月 15, 2025， https://proceedings.neurips.cc/paper/2021/hash/ba3c5fe1d6d6708b5bffaeb6942b7e04-Abstract.html
  163. Research on Deep Reinforcement Learning Control Algorithm for Active Suspension Considering Uncertain Time Delay - PubMed, 访问时间为 四月 15, 2025， https://pubmed.ncbi.nlm.nih.gov/37765884/
  164. MLOps for Low-Latency Applications: A Practical Guide - CloudFactory, 访问时间为 四月 15, 2025， https://www.cloudfactory.com/blog/mlops-for-low-latency
  165. PLC Vs. DCS – How to choose the right option for your operation, 访问时间为 四月 15, 2025， https://www.india.fujielectric.com/blog/plc-vs.-dcs-how-to-choose-the-right-option-for-your-operation
  166. Reinforcement learning methods based on GPU accelerated industrial control hardware - Fraunhofer-Publica, 访问时间为 四月 15, 2025， https://publica.fraunhofer.de/bitstreams/36c3e2e3-65fb-4317-b05d-158232b11c08/download
  167. Fiber optic amplifier has 10 microsecond response time - Control Engineering, 访问时间为 四月 15, 2025， https://www.controleng.com/fiber-optic-amplifier-has-10-microsecond-response-time/
  168. Fundamentals of real-time processing in automation and control, part 2, 访问时间为 四月 15, 2025， https://www.controleng.com/articles/fundamentals-of-real-time-processing-in-automation-and-control-part-2/
  169. Low-latency deep-reinforcement learning algorithm for ultrafast fiber lasers, 访问时间为 四月 15, 2025， https://opg.optica.org/prj/abstract.cfm?uri=prj-9-8-1493
  170. Microseconds - NI - National Instruments, 访问时间为 四月 15, 2025， https://www.ni.com/docs/en-US/bundle/labview-nxg-nodes-api-ref/page/wait-until-next-multiple-us.html
  171. [2012.09737] Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL - arXiv, 访问时间为 四月 15, 2025， https://arxiv.org/abs/2012.09737
  172. Reinforcement Learning at CERN’s accelerators - KIT Indico, 访问时间为 四月 15, 2025， https://indico.scc.kit.edu/event/3746/contributions/15477/attachments/7188/11414/RL_CERN_vkain_Feb24.pdf