One avenue for creating a generally intelligent agent is to equip it with the ability to combine its previously learned behaviours to generate new ones. Currently, most reinforcement learning approaches focus on learning task-specific skills with little consideration for learning skills that can later be composed to solve new problems immediately. In this work, we consider the setting where an agent is required to compose its existing skills to attain goals with different preferences. In particular, we show how this problem can be formulated as a contextual bandit and demonstrate how relevant algorithms can learn the optimal weighted composition of skills given a context. Experiments in a high-dimensional environment indicate that an agent operating in our framework is capable of learning these weights directly from data autonomously.