About Me

I am a postdoctoral researcher at Aalto University. My current research interests are in artificial intelligence, specifically in the possibilities for imbuing machines with traits of creativity and curiosity. I use the pronouns she/her or they/them.

Want to hear about it when I publish new work or give a talk? Join my mailing list.

At Aalto, I currently collaborate with the Autotelic Interaction Research Group and Team ARAM.

I recently co-organized the Intrinsically Motivated Open-ended Learning Workshop at NeurIPS2023.
Check out the amazing work presented by our participants!

Education

Doctor of Philosophy in Computing Science, 2023
Department of Computing Science, Faculty of Science, University of Alberta
Supervised by Dr. Patrick Pilarski
Committee Members: Dr. Martha White and Dr. Craig Chapman

Bachelor of Science in Honors Mathematics, 2014
Department of Mathematical and Statistical Sciences, Faculty of Science, University of Alberta
Graduated with First-Class Honors
Research Project: "Characterization of k-polygon graphs"
Supervised by Dr. Lorna Stewart

Contributions

Submitted or In Preparation

  1. N. M. Ady, R. Shariff, J. Günther, and P. M. Pilarski, "Five Properties of Specific Curiosity You Didn't Know Curious Machines Should Have," submitted November 30th, 2022 to Journal of Artificial Intelligence Research (JAIR). Available on arXiv at http://arxiv.org/abs/2212.00187
    Abstract
    Curiosity for machine agents has been a focus of lively research activity. The study of human and animal curiosity, particularly specific curiosity, has unearthed several properties that would offer important benefits for machine learners, but that have not yet been well-explored in machine intelligence. In this work, we conduct a comprehensive, multidisciplinary survey of the field of animal and machine curiosity. As a principal contribution of this work, we use this survey as a foundation to introduce and define what we consider to be five of the most important properties of specific curiosity: 1) directedness towards inostensible referents, 2) cessation when satisfied, 3) voluntary exposure, 4) transience, and 5) coherent long-term learning. As a second main contribution of this work, we show how these properties may be implemented together in a proof-of-concept reinforcement learning agent: we demonstrate how the properties manifest in the behaviour of this agent in a simple non-episodic grid-world environment that includes curiosity-inducing locations and induced targets of curiosity. As we would hope, our example of a computational specific curiosity agent exhibits short-term directed behaviour while updating long-term preferences to adaptively seek out curiosity-inducing situations. This work, therefore, presents a landmark synthesis and translation of specific curiosity to the domain of machine learning and reinforcement learning and provides a novel view into how specific curiosity operates and in the future might be integrated into the behaviour of goal-seeking, decision-making computational agents in complex environments.

Refereed Journal Publications

  1. C. Linke, N. M. Ady, M. White, T. Degris, and A. White, "Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study," published December 14, 2020, in the Journal of Artificial Intelligence Research. 69, 1287-1332. (Full Article) (arXiv Preprint)
    Abstract
    Learning about many things can provide numerous benefits to a reinforcement learning system. For example, learning many auxiliary value functions, in addition to optimizing the environmental reward, appears to improve both exploration and representation learning. The question we tackle in this paper is how to sculpt the stream of experience---how to adapt the system's behaviour---to optimize the learning of a collection of value functions. A simple answer is to compute an intrinsic reward based on the statistics of each auxiliary learner, and use reinforcement learning to maximize that intrinsic reward. Unfortunately, implementing this simple idea has proven difficult, and thus has been the focus of decades of study. It remains unclear which of the many possible measures of learning would work well in a parallel learning setting where environmental reward is extremely sparse or absent. In this paper, we investigate and compare different intrinsic reward mechanisms in a new bandit-like parallel-learning testbed. We discuss the interaction between reward and prediction learners and highlight the importance of introspective prediction learners: those that increase their rate of learning when progress is possible, and decrease when it is not. We provide a comprehensive empirical comparison of 15 different rewards, including well-known ideas from reinforcement learning and active learning. Our results highlight a simple but seemingly powerful principle: intrinsic rewards based on the amount of learning can generate useful behaviour, if each individual learner is introspective.
  2. J. Günther, N. M. Ady, A. Kearney, M. R. Dawson, P. M. Pilarski, "Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures," accepted February 26, 2020 to Frontiers in Robotics and AI - Computational Intelligence in Robotics. (Open-access article) (arXiv Preprint)
    Abstract
    Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well suited for robotics is that they can be learned online and incrementally through interaction with the environment. However, a remaining challenge for many prediction-learning approaches is an appropriate choice of prediction-learning parameters, especially parameters that control the magnitude of a learning machine's updates to its predictions (the learning rate or step size). To begin to address this challenge, we examine the use of online step-size adaptation using a sensor-rich robotic arm. Our method of choice, Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step sizes on a feature level; importantly, TIDBD allows step-size tuning and representation learning to occur at the same time. We show that TIDBD is a practical alternative for classic Temporal-Difference (TD) learning via an extensive parameter search. Both approaches perform comparably in terms of predicting future aspects of a robotic data stream. Furthermore, the use of a step-size adaptation method like TIDBD appears to allow a system to automatically detect and characterize common sensor failures in a robotic application. Together, these results promise to improve the ability of robotic devices to learn from interactions with their environments in a robust way, providing key capabilities for autonomous agents and robots.

Conference Presentations and Papers

  1. M. Laattala, R. Piitulainen, N. M. Ady, M. Tamariz, P. Hämäläinen, “WAVE: Anticipatory Movement Visualization for VR Dancing” The ACM CHI conference on Human Factors in Computing Systems (CHI 2024), Honolulu, Hawaiʻi, USA, May 11-14, 2024. (Short paper, accepted Jan. 19)
    Abstract
    Dance games are one of the most popular game genres in Virtual Reality (VR), and active dance communities have emerged on social VR platforms such as VR Chat. However, effective instruction of dancing in VR or through other computerized means remains an unsolved human-computer interaction problem. Existing approaches either only instruct movements partially, abstracting away nuances, or require learning and memorizing symbolic notation. In contrast, we investigate how realistic, full-body movements designed by a professional choreographer can be instructed on the fly, without prior learning or memorization. Towards this end, we describe the design and evaluation of WAVE, a novel anticipatory movement visualization technique where the user joins a group of dancers performing the choreography with different time offsets, similar to spectators making waves in sports events. In our user study (N=36), the participants more accurately followed a choreography using WAVE, compared to following a single model dancer.
  2. N. M. Ady, “Specific Curiosity is a Holistic Pursuit” The 6th International Workshop on Intrinsically Motivated Open-ended Learning (IMOL 2023), Paris, France, September 13-15, 2023. (Poster and 2-page extended abstract.)
    Abstract
    The development of machine analogs of specific curiosity, intrinsic motivation to learn something specific, has the potential to strongly benefit autonomous agents. Earlier work by Ady et al. (2022) demonstrated how three of these properties might be implemented together in a reinforcement learning agent. In this work, we highlight how the behaviour of that agent deteriorates when any one of the included properties is ablated, providing initial evidence for the interconnected nature of the properties of specific curiosity—effective learning behaviour isn't achieved via one or two properties; the properties work together.
  3. N. M. Ady and F. Rice, “Interdisciplinary Methods in Computational Creativity: How Human Variables Shape Human-Inspired AI Research” Proceedings of the 14th International Conference on Computational Creativity, ICCC'23, Waterloo, Ontario, Canada, June 23, 2023. (4-page paper and oral presentation; Best Student Paper.)
    Abstract
    The word creativity originally described a concept from human psychology, but in the realm of computational creativity (CC), it has become much more. The question of what creativity means when it is part of a computational system might be considered core to CC. Pinning down the meaning of creativity, and concepts like it, becomes salient when researchers port concepts from human psychology to computation, a widespread practice extending beyond CC into artificial intelligence (AI). Yet, the human processes shaping human-inspired computational systems have been little investigated. In this paper, we question which human literatures (social sciences, psychology, neuroscience) enter AI scholarship and how they are translated at the port of entry. This study is based on 22 in-depth, semi-structured interviews, primarily with human-inspired AI researchers, half of whom focus on creativity as a major research area. This paper focuses on findings most relevant to CC. We suggest that which human literature enters AI bears greater scrutiny because ideas may become disconnected from context in their home discipline. Accordingly, we recommend that CC researchers document the decisions and context of their practices, particularly those practices formalizing human concepts for machines. Publishing reflexive commentary on human elements in CC and AI would provide a useful record and permit greater dialogue with other disciplines.
  4. N. M. Ady, R. Shariff, J. Günther, P. M. Pilarski, "Prototyping three key properties of specific curiosity in computational reinforcement learning," 5th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM 2022), June 8-11, 2022, Brown University, Providence, RI, USA. (Poster and 5-page extended abstract.)
    Abstract
    Curiosity for machine agents has been a focus of intense research. The study of human and animal curiosity, particularly specific curiosity, has unearthed several properties that would offer important benefits for machine learners, but that have not yet been well-explored in machine intelligence. In this work, we introduce three of the most immediate of these properties -- directedness, cessation when satisfied, and voluntary exposure -- and show how they may be implemented together in a proof-of-concept reinforcement learning agent; further, we demonstrate how the properties manifest in the behaviour of this agent in a simple non-episodic grid-world environment that includes curiosity-inducing locations and induced targets of curiosity. As we would hope, the agent exhibits short-term directed behaviour while updating long-term preferences to adaptively seek out curiosity-inducing situations. This work therefore presents a novel view into how specific curiosity operates and in the future might be integrated into the behaviour of goal-seeking, decision-making agents in complex environments.
  5. N. M. Ady, "I need to know! Discomfort in Machine Curiosity," Digital Humanities Conference (DiHuCon 2021), March 15-17, 2021, University of Alberta (Virtual), Edmonton, Alberta, Canada. (Presentation and abstract.) Click here for a transcript and list of sources.
    Abstract
    Curiosity presents an apparent paradox that has long puzzled thinkers: the experience of curiosity can be downright unpleasant, a gnawing need to find out. Yet humans regularly seek out curiosity-inducing situations, as demonstrated by the popularity of cliffhanger-permeated television and books, puzzle rooms, and even crosswords. Whether we like how it feels or not, our present society values curiosity, which has lead to thriving communities of researchers interested in better understanding the mechanisms behind curiosity in humans and other animals and researchers interested in creating mechanisms so that machine intelligences can exhibit curiosity too. The purpose of this presentation is to change how you think about curiosity. I will argue how discomfort can be viewed as central to mechanisms of curiosity, helping it give rise to the properties we value when we think about the place of curiosity in schools, business, and lifelong learning.
  6. C. Linke, N. M. Ady, T. M. Degris, M. White, A. White, "Investigating Curiosity for Multi-Prediction Learning," 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), July 7-10, 2019, McGill University, Montréal, Québec, Canada. (Poster and abstract.)
    Abstract
    This paper investigates a computational analog of curiosity to drive behavior adaption in learningsystems with multiple prediction objectives. The primary goal is to learn multiple independent predictionsin parallel from data produced by some decision making policy—learning for the sake of learning. We canframe this as a reinforcement learning problem, where a decision maker’s objective is to provide trainingdata for each of the prediction learners, with reward based on each learner’s progress. Despite the varietyof potential rewards—mainly from the literature on curiosity and intrinsic motivation—there has been littlesystematic investigation into suitable curiosity rewards in a pure exploration setting. In this paper, weformalize this pure exploration problem as a multi-arm bandit, enabling different learning scenarios to besimulated by different types of targets for each arm and enabling careful study of the large suite of potentialcuriosity rewards. We test 15 different analogs of well-known curiosity reward schemes, and compare theirperformance across a wide array of prediction problems. This investigation elucidates issues with severalcuriosity rewards for this pure exploration setting, and highlights a promising direction using a simplecuriosity reward based on the use of step-size adapted learners.
  7. J. Günther, A. Kearney, N. M. Ady, C. Sherstan, M. R. Dawson, and P. M. Pilarski, "GVFs: General Value Freebies," 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), July 7-10, 2019, McGill University, Montréal, Québec, Canada. (Poster and abstract.)
    Abstract
    Machine learning offers the ability for machines to learn from data and improve their performance on a given task. The data used in learning is usually provided either in terms of a predesigned data set or as sampled through interaction with the environment. However, there is another oft-forgotten source of data available for machines to learn from: the learning process itself. As algorithms learn from data and interact with their environment, learning mechanisms produce a continuous stream of data in terms of errors, parameters changes, updates and estimates. These signals have great potential for use in learning and decision making. In this paper, we investigate the utility of such “freebie” signals that are produced either as the output of learning or due to the act of learning, i.e., updates to weights and learning rates. Specifically, we implement a prediction learner that models its environment via multiple General Value Functions (GVFs) and deploy it within a robotic setting. The first signal of interest that we study is one known as the Unexpected Demon Error (UDE), which is closely related to the Temporal-Difference (TD) error and can be tied to the notion of surprise. Detecting surprise reveals important information not only about the learning process but also about the environment and the functioning of the agent within its environment. The second type of signal that we investigate is the agent's learning step size. For this purpose, a vectorized step-size adaptation algorithm is used to update the step sizes over the course of learning. Observing the step-size distribution over time appears to allow a system to automatically detect and characterise common sensor failures in the physical system. We suggest that by adding introspective signals such as UDE and step sizes analysis to the available data, autonomous and long-lived agents can become better aware of their interactions with the environment, resulting in a superior ability to make decisions.
  8. J. Günther, A. Kearney, N. M. Ady, M. R. Dawson, P. M. Pilarski, “Meta-learning for Predictive Knowledge Architectures: A Case Study Using TIDBD on a Sensor-rich Robotic Arm,” Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13–17, 2019, IFAAMAS, pp. 1967–1969. (Extended abstract and poster.) (PDF)
  9. N. M. Ady and P. M. Pilarski, “Comparing Reinforcement Learning Methods for Computational Curiosity through Behavioural Analysis,” 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), University of Michigan, Ann Arbor, Michigan, USA, June 11-14, 2017. (Poster and abstract.)
  10. N. M. Ady and P. M. Pilarski, “Unifying Curious Reinforcement Learners,” Designing for Curiosity: An Interdisciplinary Workshop, ACM CHI Conference on Human Factors in Computing Systems (CHI 2017), Denver, Colorado, USA, May 6-11, 2017. (Poster and extended abstract.)
  11. N. M. Ady and P. M. Pilarski, “Domains for Investigating Curious Behaviour in Reinforcement Learning Agents,” 11th Women in Machine Learning (WiML) Workshop, Barcelona, Spain, December 5, 2016. (Poster and abstract.)
  12. N. Ady and F. Rice, “A Disciplinary Divide: Does Discipline-Specific Coaching Make a Difference?” Canadian Writing Centres Association / L’Association Canadienne des Centres de Rédaction Conference: Writing without borders, Brockville, Ontario, Canada, May 23, 2014. (Abstract and oral presentation.)
    Abstract
    To increase the efficacy of the tutoring program at the Centre for Writers, we reviewed the literature regarding matching writing tutors from the same discipline as their clients versus matching clients with tutors of different disciplines. We also used surveys of our clients and their performance in their classes before and after tutoring to perform our own analysis and provide insight into the observed advantages and disadvantages of each matching.

Multidisciplinary Symposia Presentations

  1. N. M. Ady and P. M. Pilarski, “5 Properties You Didn't Know Curious Machines Should Have,” Amii Research Day, Alberta Machine Intelligence Insitute (virtual), June 15, 2021. (Poster.)
  2. J. Ventura, N. M. Ady, and P. M. Pilarski, “An Exploration of Artificial Curiosity and Reinforcement Learning in a Simple Robot,” WISEST Poster Session, University of Alberta, 2017. https://doi.org/10.7939/R36W96Q00 (Poster.)
  3. N. M. Ady and P. M. Pilarski, “Behaviour of Curious Reinforcement Learners Faced with Varying Reward,” 2017 CRA-W Grad Cohort Workshop Poster Session, Washington D.C., USA, April 7, 2017. (Poster and abstract.)
  4. N. M. Ady, “Undergraduate Preparation for Writing in the Discourse Community of Mathematics” Canadian Undergraduate Mathematics Conference, Carleton University, Ottawa, Ontario, Canada, July 2-5, 2014. (Oral presentation and abstract.)
    Abstract
    Upon completion of their undergraduate degree, are aspiring mathematicians prepared for the writing needed in mathematical academia? Mathematicians in academic positions must communicate a variety of ideas to a variety of audiences, and so must learn the skills needed to do so effectively. This talk considers what those writing skills are and their development in an undergraduate program. The talk looks at current undergraduate mathematics education in writing. A discussion of discourse style and writing culture in mathematics allows an understanding of the gap facing students when they graduate. Given the importance of good writing and communication to the math community, the conclusions include improvements that might be made both for the sake of undergraduates and to improve the strength of the mathematical discourse community.
  5. N. M. Ady, A. St. Arnaud, and L. K. Stewart, “Recognition of k-polygon graphs,” 3rd Annual University of Alberta Undergraduate Research Symposium, University of Alberta, Edmonton, Alberta, Canada, November 22, 2013. (Poster.)

Technical Reports

  1. N. M. Ady, "Curious Actor-Critic Reinforcement Learning with the Dynamixel-bot," University of Alberta, 2017. https://doi.org/10.7939/R3B853Z7S (Technical report, 7 pages.)
    Abstract
    Curiosity is a crucial, but not yet well-understood component of intelligence and a better understanding of existing models may lead to a better understanding of curiosity as a whole. In this work, we present a physical robot implementation of the basic curiosity loop introduced by Gordon and Ahissar in 2012. In the same way that Gordon and Ahissar produced a rough simulation of a rat’s whiskers, this work presents a physical model using two servos to create the whisking actions.
  2. N. M. Ady, "Parameter Screening for Curious Reinforcement Learner Motivated by Unexpected Error," University of Alberta, 2017. https://doi.org/10.7939/R3G15TS0P (Technical report, 8 pages + appendix.)
    Abstract
    Curiosity is a critical component of intelligence. One method of motivating curious behaviour in computational systems is to use reinforcement learning to learn which decisions maximize the amount of unexpected error observed by a predictive component. However, reinforcement learning algorithms for prediction and control require the system designer to set multiple parameters, and it is unknown how such a curious system’s behaviour might vary depending on parameter settings. Eight parameters (one learning rate, continuation probability, trace decay parameter for both prediction and control, 'epsilon' (the probability of a random action for epsilon-greedy control) and beta-naught parameter for computation of White’s (2015) unexpected error) were tested in an inscribed central composite experimental design. The response variable was the return. We found that the linear effects on return for epsilon, the learning rate for control, the continuation probability for prediction, and the beta-naught parameter for unexpected error were significant, along with the quadratic interactions between epsilon and beta-naught, epsilon and the continuation probability for prediction, beta-naught and the continuation probability for prediction, and the learning rate and continuation probability for prediction.

Invited Presentations

  1. “Machine curiosity: What should we expect of curious AI—once it exists?”
    Mount Royal University, Calgary, Canada, November 23, 2023.
  2. “Curiosity as Explored by AI Scientists”
    University of Edinburgh, Edinburgh, United Kingdom, June 6, 2019. (Full-length seminar.)
  3. “Curiosity in Machine Intelligence”
    Amii’s AI Meet-up (for Alberta Machine Intelligence Institute), Startup Edmonton, Edmonton, Alberta, Canada, January 22, 2019. (Half-length seminar.)
  4. “An Overview of Computational Curiosity in Reinforcement Learning”
    Kindred AI, Toronto, Ontario, Canada, July 11, 2018. (Half-length seminar.)
  5. “Overview of Curiosity in Computational Reinforcement Learning”
    Princeton Neuroscience Institute and Psychology Department, Princeton University, Princeton, New Jersey, June 28, 2018. (Half-length seminar.)
  6. “Writing Centres: Becoming a better writer from the comfort of your own campus”
    Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada, October 2016.
  7. “Curious About Computational Curiosity? Introduction and Experiments”
    Reasoning and Learning Lab, McGill University, Montréal, Québec, Canada, August 9, 2016. (Full-length seminar.)
  8. “On Being a Graduate Student in Computing Science”
    Microsoft Store, West Edmonton Mall, Edmonton, Alberta, Canada, April 23, 2016.
  9. “License Plate Recognition” co-presented with T. Griffith and A. Neufeld
    Edmonton Office of Traffic Safety, Edmonton, Alberta, Canada, April 17, 2014.

Selected Employment History

Research Positions

Present
Postdoctoral Researcher
Autotelic Interaction Research Group
Department of Computer Science, Aalto University, Espoo, Finland
Helsinki Insitute for Information Technology
2016 - 2023
Ph.D. Candidate and Graduate Research Assistant
Supervisor: Dr. Patrick Pilarski
Department of Computing Science, University of Alberta, Edmonton
Reinforcement Learning and Artificial Intelligence Lab Alberta Machine Intelligence Institute Bionic Limbs for Improved Natural Control
Summer 2015
Systems Administrator - Student Co-op
Government of Canada, Ottawa
Fall 2014
Scientist - Student Co-op
Government of Canada, Ottawa
Summer 2014
Research Assistant (Wireless Networks)
Supervisor: Dr. Mike MacGregor
Department of Computing Science, University of Alberta, Edmonton
Summer 2013
Research Assistant (Graph Algorithms)
Supervisor: Dr. Lorna Stewart
Department of Computing Science, University of Alberta, Edmonton

Academic Teaching Positions

Winter 2013 to Winter 2016
Writing Tutor and Workshop Instructor
Centre for Writers, University of Alberta, Edmonton
Winter 2016, Fall 2015, Winter 2015
Teaching Assistant for CMPUT 174
Introduction to the Foundations of Computation I
Department of Computing Science, University of Alberta, Edmonton
Winter 2014
Teaching Assistant for CMPUT 474
Formal Languages, Automata and Computability
Department of Computing Science, University of Alberta, Edmonton