Continuous Dynamic Drill-Off Test Whilst Drilling Using Reinforcement Learning in Autonomous Rotary Drilling System

Abstract

Abstract In the development of autonomous downhole drilling systems, decision-making in the selection of optimized operating parameters has become one of the technical difficulties. Traditionally, the driller performs a trial-and-error approach in search of optimal parameters, which is now less effective and non-sustainable to the changing drilling environment. This paper presents a decision-making process using reinforcement Q-learning algorithms that can act as real-time optimization algorithms for selecting an optimal operating parameter for rotary drilling systems using Q-learning on experimental published data from the literature. The reinforcement learning framework is a stochastic approximate dynamic programming, with varying estimation techniques for goal-directed sequential learning from interaction paradigms. First, a Markov Decision Process (MDP) is established by analyzing agent exploration and exploitation of possible actions taken in an environment. Second, the state set and action set are designed by the synthesized consideration of surface operating parameters from the published data within the range of operational limit. Then, sequentially, at each timestep, the agent takes an action (e.g., changing rotary speed or changing axial force) that makes the environment (formation) transition from one state to another. Consequently, the agent receives a reward (e.g., distance drilled) before taking the next action. Furthermore, a recursive reinforcement Q-learning algorithm is developed mainly based on the reward function and update function. Analysis of experimental data on drilling was implemented for five states of axial force parameters with five feed rate decisions on each of the states, whilst having distance of a hole drilled as a reward. The proposed optimization model computed using value iteration showed that following Decision 2 yielded the best result. The analysis results also revealed that the optimal value function was reached irrespective of the initial state conditions. The agent's objective is to learn policy mapping from states to actions such that the agent's cumulative reward (footage drilled) is maximized. The result of this research could be used as a decision-making tool in drilling operations that provides an engineered approach for optimal operating parameter selection and improvement in the efficiency of the drilling process in terms of cost and time.