Large distributed computer systems have been successfully employed to solve modern scientific problems that were previously untenable. With the advent of low cost hardware, high speed computers and sizable storage resources are pervasive and often notably underutilized. Modern opportunistic middleware systems seek to congregate the potential of these existing systems into a powerful resource for scientific computation. Within distributed computing, the study of these systems considers the ability of middleware to respond to heterogeneous or low reliability environments, as well as the need to obtain high performance or guaranteed quality of service. In various forms such as Internet computing, desktop grids, and even ``big iron' grids, this philosophy realizes the genuinely unpredictable nature of large scale computing projects. The development effort in this field takes two important forms: it attempts to build aggregate systems that are more reliable than their underlying components, easier to utilize in concert, and effectively unified administratively; it attempts to provide performance that is timely and predictable. In this dissertation, the need to construct reliable, high performance storage from loosely trusted, low reliability components is addressed. We investigate methods to catalog and maintain large networks of storage components and present them to scientific users as a unified resource. The methods developed in this area resulted in the implementation of the GEMS replica management system. The need to obtain good, reliable, predictable performance when performing a new computation is also addressed. We investigate methods to obtain job runtime estimate information and use this knowledge to establish probabilistic guarantees on the completion time of new computation. The methods developed in this area resulted in the implementation of the East scheduling simulator. A common idea underlines both techniques: the use of a higher level managing software system that operates around an internal model of the controlled resources. This model is both queried for historical and predictive information and used by a middleware grid overdrive controller to make intelligent management decisions that satisfy user requirements. The design of the controller borrows from techniques used in a variety of disciplines including mathematical modelling, system theory, and autonomic systems to manage user jobs and data. The result is a synthesis of existing building blocks and new abstractions that results in efficiency, practicality, and scientific benefit.