The systematic series of advances in semiconductor device manufacturing over the last 70 years has enabled previously unforeseen applications and allowed semiconductor device technologies to become a hidden yet central part of everyday modern life. In the past, the computational and data storage demands of our ever- expanding digital information ecosystem have been primarily fulfilled through scaling silicon based complementary-metal-oxide-semiconductor (CMOS) technologies. However, in order to fulfill the demands of future computing systems, the continued scaling of current semiconductor device technologies alone will not suffice. Instead, a deep holistic integration is required, where devices, circuits, architectures, and algorithms are co-designed and co-optimized, rather than seeking drop in replacements for current devices technologies. In this light, this thesis explores the response of emerging devices based on collective phenomena, including insulator-to-metal phase transitions and ferroelectrics, with the goal to intimately understand the underlying electrically driven dynamics and how they can be harnessed to either engineer or augment, new and existing hardware primitives. This dissertation describes key enabling device characteristics, their physical origin, and scalability, including abrupt switching, multi-domain behavior, and stochastic response. This work reveals how fundamental stochastic dynamics in phase transition materials can be exploited for computation, allowing new designs of true random number generators as well as stochastic neuron cells tailored to neuromorphic computing architectures and algorithms. Further, this work discusses the design of an analog synaptic weight cell using the natural multi- domain dynamics and non-volatility of ferroelectric field-effect-transistors for accelerating neuromorphic workloads compared to existing CMOS technologies. The dissertation concludes by assessing future research directions, including remaining opportunities and challenges presented by phase transition and ferroelectric devices.