PhD Studentship: Leveraging Reinforcement Learning to Improve Long-term Planning in Generative Models

This 3.5 year PhD project is fully funded and home students, and EU students with settled status, are eligible to apply. The successful candidate will received an annual tax free stipend set at the UKRI rate (£19,237 for 2024/25) and tuition fees will be paid. We expect the stipend to increase each year.

Generative models, such as large language models (LLMs), have exhibited remarkable success in generating coherent text, mimicking human-like conversations, and completing complex language tasks such as translation and summarization. However, despite their powerful generative capabilities, these models often struggle with tasks requiring deeper reasoning, logical coherence, and long-term planning. These limitations hinder their applicability in domains where reasoning and long-term decision-making are crucial, such as legal analysis, scientific discovery, and industrial planning tasks.

Daniel Kahneman's concept of Thinking, Fast and Slow distinguishes between two modes of cognitive functioning: System 1 (fast, intuitive, and automatic thinking) and System 2 (slow, deliberate, and effortful reasoning). While LLMs currently excel at System 1-type tasks (i.e., surface-level and associative tasks), they fall short in System 2-type reasoning (i.e., logical, long-term, and multi-step problem-solving).

[Sys2RL] This research proposes leveraging Reinforcement Learning (RL) to enhance the reasoning capabilities of generative models by introducing a training regime inspired by the Thinking, Fast and Slow paradigm. Recently, the use of RL has been shown to significantly improve the performance of LLMs. The goal of this proposal is to improve their System 2-like abilities by systematically integrating reinforcement learning techniques, encouraging models to reason more deeply when required.

Applicants should have, or expect to achieve, at least a 2.1 honours degree or a master’s in a relevant science or engineering related discipline. Applicants should have strong background in Machine Learning and Deep Learning.

To apply, please contact the supervisors; Dr Mingfei Sun - mingfei.sun@manchester.ac.uk and Prof Samuel Kaski - samuel.kaski@manchester.ac.uk. Please include details of your current level of study, academic background and any relevant experience and include a paragraph about your motivation to study this PhD project.

Advert information