Building blocks for commodity augmented reality-based molecular visualization and modeling in web browsers Building blocks for commodity augmented reality-based molecular visualization and modeling in web browsers Luciano A. Abriata1,2 1 École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland 2 Swiss Institute of Bioinformatics, Lausanne, Switzerland ABSTRACT For years, immersive interfaces using virtual and augmented reality (AR) for molecular visualization and modeling have promised a revolution in the way how we teach, learn, communicate and work in chemistry, structural biology and related areas. However, most tools available today for immersive modeling require specialized hardware and software, and are costly and cumbersome to set up. These limitations prevent wide use of immersive technologies in education and research centers in a standardized form, which in turn prevents large-scale testing of the actual effects of such technologies on learning and thinking processes. Here, I discuss building blocks for creating marker-based AR applications that run as web pages on regular computers, and explore how they can be exploited to develop web content for handling virtual molecular systems in commodity AR with no more than a webcam- and internet-enabled computer. Examples span from displaying molecules, electron microscopy maps and molecular orbitals with minimal amounts of HTML code, to incorporation of molecular mechanics, real-time estimation of experimental observables and other interactive resources using JavaScript. These web apps provide virtual alternatives to physical, plastic-made molecular modeling kits, where the computer augments the experience with information about spatial interactions, reactivity, energetics, etc. The ideas and prototypes introduced here should serve as starting points for building active content that everybody can utilize online at minimal cost, providing novel interactive pedagogic material in such an open way that it could enable mass-testing of the effect of immersive technologies on chemistry education. Subjects Bioinformatics, Computational Biology, Human-Computer Interaction, Multimedia Keywords Molecular modeling, Integrative modeling, Virtual reality, Augmented reality, Chemistry, Education, Molecular visualization INTRODUCTION For a long time it has been suggested that visual immersive analytics based in virtual reality (VR), augmented reality (AR) and other advanced forms of human-computer interactions have enormous potential in assisting thinking processes in scientific research and in education, especially in areas of science that deal with abstract objects, objects much smaller or larger than human dimensions, objects that are hard to acquire and handle due to high costs, scarcity or fragility, and very large amounts of data (O’Donoghue et al., 2010; Matthews, 2018; Krichenbauer et al., 2018; Sommer et al., 2018). Chemistry and How to cite this article Abriata LA. 2020. Building blocks for commodity augmented reality-based molecular visualization and modeling in web browsers. PeerJ Comput. Sci. 6:e260 DOI 10.7717/peerj-cs.260 Submitted 11 April 2019 Accepted 22 January 2020 Published 17 February 2020 Corresponding author Luciano A. Abriata, luciano.abriata@epfl.ch Academic editor James Procter Additional Information and Declarations can be found on page 18 DOI 10.7717/peerj-cs.260 Copyright 2020 Abriata Distributed under Creative Commons CC-BY 4.0 http://dx.doi.org/10.7717/peerj-cs.260 mailto:luciano.�abriata@�epfl.�ch https://peerj.com/academic-boards/editors/ https://peerj.com/academic-boards/editors/ http://dx.doi.org/10.7717/peerj-cs.260 http://www.creativecommons.org/licenses/by/4.0/ http://www.creativecommons.org/licenses/by/4.0/ https://peerj.com/computer-science/ structural biology are examples of such disciplines where AR and VR have been attributed high potential in education and research, by providing hybrid physical/computational interfaces to handle and explore virtual molecules in real 3D space augmented with real-time overlay of information from databases and calculations. However, the actual impact of immersive technologies on teaching, learning and working in chemistry still requires deep evaluation (Fjeld & Voegtli, 2002; Pence, Williams & Belford, 2015; Matthews, 2018; Bach et al., 2018; Yang, Mei & Yue, 2018). Such evaluation has progressed very slowly due to the complex software setups and the specialized hardware needed, which limit availability, reach, adoption, and thus testing. Indeed this limitation is shared more broadly with other potential applications of AR and VR in science, which so far “[…] remain niche tools for scientific research” (Matthews, 2018). In the last decade several groups have been studying ways to achieve immersive environments for chemistry and structural biology using VR and AR (Gillet et al., 2004, 2005; Maier, Tönnis & GudrunKlinker, 2009; Maier & Klinker, 2013; Hirst, Glowacki & Baaden, 2014; Berry & Board, 2014; Martínez-Hung, García-López & Escalona-Arranz, 2016; Vega Garzón, Magrini & Galembeck, 2017; Balo, Wang & Ernst, 2017; Goddard et al., 2018a, 2018b; Wolle, Müller & Rauh, 2018; O’Connor et al., 2018; Ratamero et al., 2018; Müller et al., 2018; Stone, 2019). Such interfaces allow handling molecules over 6 degrees of freedom and with both hands, in immersive 3D. They are expected to overcome the limitations of traditional software based on screen, mouse and keyboard, thus enabling more intuitive, fluid exploration of molecular features and data. At the time of writing, most of these works are not complete AR or VR versions of fully-fledged programs, but rather prototypes, proofs of concept and case studies on how humans interact with virtual molecules in AR or VR. Notable highlights moving towards complete immersive molecular visualization and modeling programs are the new rewrite of Chimera, ChimeraX, which was optimized for the new GPUs and incorporates support for VR experiences (Goddard et al., 2018b), VRmol (Xu et al., 2019), new VR plugins for the VMD molecular graphics program (Stone, 2019), and a few commercial programs like Autodesk’s Molecule Viewer (https://autodeskresearch.com/groups/lifesciences) or Nanome (https://nanome.ai/), all with interfaces specifically tailored for VR. Most of these works suffer from two severe limitations. First, all but a few exceptions require hardware such as head-mount displays (helmets, headsets or goggles like MS Hololens, Oculus Rift, HTC Vibe, etc.) or immersive installations with large surround screens plus 3D-handheld input devices and the corresponding computer and GPU hardware. The few remarkable exceptions are prototypes using ordinary webcam-enabled smartphones (Balo, Wang & Ernst, 2017) or laptops (Gillet et al., 2004). The second limitation is the need of specialized programs that often must be correctly interfaced to couple the different components required for an AR or VR experience, that is, tracking limbs, handheld devices or AR markers, then running calculations on molecular data, and finally displaying results and molecular graphics on screen (see Ratamero et al., 2018; Gillet et al., 2005). Some of these programs are only compatible with specific VR devices and many are not free software. Overall, then, despite the dropping costs, access to these tools still requires investment in the order of hundreds to low-thousand American dollars per Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 2/23 https://autodeskresearch.com/groups/lifesciences https://nanome.ai/ http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ user, and software interfacing that may not be available to lay students and teachers. It is therefore unlikely that VR will achieve the ideal of one device per student within the next few years. To date, these solutions are not widely used across the world, and their costs make them totally out of reach for educational centers in developing countries. Additionally, most current setups are limited to VR, but it has been shown that AR is more suited for educational purposes because by not occluding the view of the user’s own limbs, it results in better motion coordination and object handling than VR (Sekhavat & Zarei, 2016; Gaffary et al., 2017; Krichenbauer et al., 2018). Furthermore, in AR the view of the world is not obstructed thus allowing students and teachers to interact more easily. The purpose of this work is to demonstrate that client-side web technologies have matured enough to enable web pages for AR-based molecular visualization and modeling running just on web browsers in regular webcam-equipped computers. This enables the easy creation of immersive educational material that can be employed by students and teachers at very affordable costs and with very simple setups. All they must do is access a webpage, enable webcam activity in the browser, and show to the webcam a printed AR marker on which the molecules will be displayed. From the developer side, the code is made simple thanks to a large number of libraries; in fact visualization-only applications are achievable just with HTML code while interactivity can be incorporated through custom JavaScript. This article is organized in two parts. Part 1 provides a practical overview of the main building blocks available as of 2019 to program AR apps in web pages, with a focus on ways to achieve molecular visualization and modeling. It also briefly explores ways to handle gesture- and speech-based commands, molecular mechanics, calculation of experimental observables, concurrent collaboration through the world wide web, and other human- computer interaction technologies available in web browsers. Part 2 of the article showcases prototype web apps for specific tasks of practical utility in pedagogical and research settings. These web apps are based on open, commodity technology that requires only a modern browser “out of the box”, so educators, students and researchers are free to try out all these examples on their computers right away by following the provided links. PART 1: OVERVIEW OF BUILDING BLOCKS FOR IMMERSIVE MOLECULAR MODELING IN WEB BROWSERS Virtual and augmented reality At the core of immersive experiences are visualizations based on VR or AR methods. While VR is probably best experienced with VR goggles to suppress any side view of the real world, AR is more amenable to devices like desktop or laptop computers, tablets and smartphones, making it better suited for commodity solutions, and has the additional advantages outlined in the introduction. The methods presented in this article make use of marker-based AR in a web-based setup that functions as an “AR mirror” where the user sees him/herself with the virtual objects, here molecules, overlaid on the markers he/she holds in their hands (Figs. 1A and 1B). The following descriptions focus on this technology and how it can be implemented in web apps by using HTML, CSS and Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 3/23 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ JavaScript code as standard web pages that are hosted at servers but run entirely on local computers (Fig. 1C). The WebGL API provides powerful 2D and 3D graphing capabilities using GPU resources, in a format fully integrable with other HTML elements, APIs and JavaScript libraries, without the need of plug-ins; and is highly standardized across browsers. It is thus possible to couple all elements required to build up an AR experience directly inside the web page source code. A handful of JavaScript libraries exploit WebGL to facilitate rendering of 3D scenes, Three.js (https://threejs.org/) being probably the most widely used. In turn, tools like A-Frame (https://aframe.io/) provide entity component system frameworks that wrap Three.js into HTML language tags through polyfills, greatly facilitating the development of AR and VR scenes. The examples presented in Part 2 showcase either direct use of Three.js or of Three.js through A-Frame for AR. These libraries/frameworks can be used either by (i) loading pre-made models of the molecular Figure 1 Commodity augmented reality in web browsers. (A) AR markers “Hiro” and “Kanji” are built into AR.js and its A-Frame wrap. Figure S1 shows them ready to print at various sizes. Custom markers including cube markers are also implementable. (B) In the proposed web apps the AR library feeds coordinate information to the graphics libraries and other algorithms. (C) The web pages must be hosted in servers compatible with the https protocol, but their content then runs exclusively on the computer. Full-size DOI: 10.7717/peerj-cs.260/fig-1 Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 4/23 https://threejs.org/ https://aframe.io/ http://dx.doi.org/10.7717/peerj-cs.260/supp-1 http://dx.doi.org/10.7717/peerj-cs.260/fig-1 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ systems in formats like Wavefront’s OBJ+MTL files exported straight out of molecular visualization programs like VMD (Humphrey, Dalke & Schulten, 1996), UnityMol (Doutreligne et al., 2014; Wiebrands et al., 2018), ChimeraX (Goddard et al., 2018b), etc., possibly further edited with a program like Blender as in Martínez-Hung, García-López & Escalona-Arranz (2016); or (ii) employing 3D primitives (like spheres, cylinders, etc.) to draw the molecular systems from scratch with their atomic coordinates. Use of Wavefront models is much simpler (only a few lines of HTML code to load and display objects) and allows seamless display of any element exported from molecular graphics programs, including not only different representations of molecules but also isosurfaces as those needed to visualize volumetric data describing molecular orbitals, electron microscopy maps, etc. On the other hand, using 3D primitives requires larger pieces of code to create all 3D objects from atomic coordinates, but in turn allows for fine dynamic control of shapes, sizes, colors, and positions, which are key to interactivity. Another downside of Wavefront models is that files can become quite large for high-resolution textures. Object detection and tracking The other key component required for AR and VR is a means to detect and track objects or parts of the user’s body such as his/her hands, in order to manipulate virtual objects. Applications using ad hoc hardware use sensors and cameras that track the user’s position in space and handheld controllers that the user sees as virtual tweezers or hands, to directly move objects in space. For commodity AR/VR in web browsers, solutions rely on JavaScript versions of tracking libraries that implement computer vision algorithms through the webcam feed, like ARToolKit (Kato, 1999) among other similar solutions. These libraries track user-defined 2D markers (like those in Fig. 1A; Figs. S1 and S2) in space and make their computed coordinates available to the graphics algorithms. Two especially interesting libraries, used in many of the examples presented in Part 2, are AR.js (https://github.com/jeromeetienne/AR.js) and its A-Frame wrap (https://jeromeetienne. github.io/AR.js/aframe/). These enable highly simplified AR/VR, even using exclusively HTML code for simple developments. It is important to note that in marker-based AR different viewers looking at the same physical marker receive different perspectives of it and hence of the rendered virtual object, just as if it was a real object in real space (Fig. S3). This easily enables multi-user AR in a common room, as would be useful in a classroom setting where students and teachers look at the same virtual molecule. An alternative to traditional marker-based AR should in principle be possible by using a plain hand tracking JavaScript library like Handtracking.js. Another slightly more expensive approach but possibly better in tracking performance is using a device like the infrared-based Leap Motion Controller (https://www.leapmotion.com/), which includes a JavaScript library to feed positional information from the device into the web app. Unfortunately, however, there are currently no ready-to-use libraries that couple these input tools to WebGL graphics, so their use would require further developments. Current JavaScript libraries for computer vision allow even more complex object tracking. One interesting example is gesture recognition by the WebGazer.js library, which Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 5/23 http://dx.doi.org/10.7717/peerj-cs.260/supp-1 http://dx.doi.org/10.7717/peerj-cs.260/supp-1 https://github.com/jeromeetienne/AR.js https://jeromeetienne.github.io/AR.js/aframe/ https://jeromeetienne.github.io/AR.js/aframe/ http://dx.doi.org/10.7717/peerj-cs.260/supp-1 https://www.leapmotion.com/ http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ analyzes face features to estimate where on the screen the user is looking at (Papoutsaki et al., 2016). In molecular visualization, this can be used for example to automatically rotate regions of interest to the front of the field of view, as shown in Fig. S4. Speech-based interfaces A speech-based interface can be very useful for situations in which the user’s hands are busy holding objects, as in AR/VR applications. In-browser speech recognition APIs enable implementation of speech recognition very easily, especially through libraries like Annyang (Ater, 2019) which is used in some of the examples of this article. These libraries usually allow working in two modes, one where the browser waits for specific commands (while accepting variable arguments) and one where the browser collects large amounts of free text that are then made available to the environment. The former allows direct activation of functions without the need for the user to click on the screen. The second option opens up the possibility of automatically detecting subjects, actions and concepts that are fed to artificial intelligence routines, or just predefined rules, that the computer will analyze in background. For example, when two users are discussing the interaction surface between two proteins and mention certain residues, the computer could automatically mine NIH’s PubMed repository of papers for mentions of said residues. This may seem far-fetched, but is essentially the same technology that underlies automatic advertising and suggestions based on users’ various inputs and usage statistics in software and websites. The problem of intelligently suggesting chemical and biological information related to a topic or object has already been addressed for some time, for example in information augmentation tools like Reflect (Pafilis et al., 2009) and advanced text-mining tools (Rebholz-Schuhmann, Oellrich & Hoehndorf, 2012). The evolution of science-related standards are very important in this regard, formats and content for the semantic web (Hendler, 2003) and of machine-readable scientific databases and ontologies that standardize knowledge and data. Intensive calculations As recently reviewed in a dedicated issue of Computing and Science in Engineering (DiPierro, 2018), JavaScript has become very powerful by including language subsets specialized for speed, optimized just-in-time compilation, methods to program background scripts, and libraries to perform multi-core and on-GPUs computing to accelerate intensive calculations. It is in fact now possible to transpile from C/C++ and other languages directly to JavaScript retaining close-to-native execution speeds, allowing web browsers to support quite complex data analysis and visualization directly in web pages (Abriata et al., 2017; Abriata, 2017a). This opens up the possibility of simulating molecular mechanics and experimental data, performing numerical data analysis and even handling data in neural networks, directly inside the molecular modeling web app to enable real-time feedback as the user manipulates the molecular systems. Some of the prototypes presented in Part 2 include such examples. Rather than manually implementing calculations, a number of libraries are now available for specific applications that help to save code writing time and bring the Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 6/23 http://dx.doi.org/10.7717/peerj-cs.260/supp-1 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ additional advantage of being developed by specialists (Abriata, 2017a). One particularly useful library in the scope of this paper is Cannon.js (https://schteppe.github.io/cannon.js/), an engine for simulating rigid body mechanics that integrates smoothly with Three.js and A-Frame. These engines use numerical methods to propagate motions of solid bodies connected by springs and other physical constraints in an approximate but efficient way, thus adding realistic physics to the 3D bodies of AR and VR worlds. Although these engines do not account for all the forces and phenomena of the atomic realm, such as electrostatic interactions and quantum phenomena, they are useful in certain real scenarios of molecular modeling. For example, the Integrative Modeling Platform software (used to put together partial structures into bigger assemblies, driven by experimental data) includes one such kind of physics engine for molecule-molecule docking (Russel et al., 2012). Furthermore, implementation of more complex force fields is certainly possible, as exemplified by a JavaScript transpilation of the OpenMD molecular dynamics engine (Jiang & Jin, 2017). Further building blocks Any other technology that facilitates interaction with the computer within a 3D environment, either to deliver or obtain information, might be of use. For example, haptic feedback would be desirable to confer a physical feel of interactions and clashes as in (Wollacott & Merz, 2007; Stocks, Hayward & Laycock, 2009; Stocks, Laycock & Hayward, 2011; Matthews et al., 2019). Achieving a good experience in haptic feedback currently requires specialized devices, and is an area of active research (Bolopion et al., 2009, 2010), therefore it does not fit with the commodity hardware criteria outlined in the introduction. Other rudimentary ways to achieve sensory feedback include exploiting built-in vibration devices and touch-pressure sensing in touch screens, both handled by JavaScript APIs. Finally, a particularly interesting aspect of software running on web browsers is the ease with which different users can connect to each other, just over the internet. Web apps can exploit web sockets to achieve direct browser-to-browser links over which data can be transmitted freely, with a server only intervening to establish the initial connection (Pimentel & Nickerson, 2012). For example, two or more users can concurrently work on a JSmol session by sharing just mouse rotations and commands, appearing on all other users’ screens with a minor delay (Abriata, 2017b). Such collaborative working technology could be adapted to complex immersive environments to allow multiple users to work on chemical problems at a distance, essential for scientific collaborations, demonstrations, and online teaching (Lee, Kim & Kang, 2012). PART 2: PROTOTYPE WEB APPS SHOWCASING EXAMPLE APPLICATIONS This section presents example AR web apps compatible with major web browsers in modern computers, introducing features of increasing complexity. All examples were verified to run out of the box in multiple web browsers on Windows, Linux and MacOS operating systems, in laptop and desktop computers. All the examples are accessible through links in Table 1 and at https://lucianoabriata.altervista.org/papersdata/tablepeerjcs2019.html, which Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 7/23 https://schteppe.github.io/cannon.js/ https://lucianoabriata.altervista.org/papersdata/tablepeerjcs2019.html http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ further contains links to demo videos. To run these examples the user needs to print the Hiro, Kanji or cube markers as needed in each case (Fig. 1A; Figs. S1 and S2, and links on web pages). For simpler handling, flat markers (Hiro and Kanji) may be glued on a flat surface mounted on a device that can be easily rotated from the back, such as a small shaft perpendicular to the marker plane. The cube marker is printed in a single page, folded and possibly glued to a solid cube made of wood, plastic, rubber or similar material. Readers interested in the inner workings and in developing content can inspect the source code of each webpage (Ctrl+U or Cmd+U in most browsers). Several recommendations and basic troubleshooting for developers and users are given in Table 2. Introducing web browser-based AR for visualization The simplest way to achieve AR in web pages consists in displaying on the AR markers representations exported from programs like VMD (Humphrey, Dalke & Schulten, 1996) in Wavefront (OBJ+MTL) format. This can be achieved with a few lines of HTML code thanks to libraries like AR.js for A-Frame, enabling the very easy creation of content for displaying any kind of object handled by the exporting program. Figure 2A exemplifies this with a small molecule, 2-bromo-butane, shown as balls and sticks. This small molecule is chiral at carbon 2; the Hiro marker displays its R enantiomer while the Kanji marker displays the S enantiomer, both rendered from the same pair of OBJ+MTL files but scaled as required for chirality inversion. Figure 2B shows on the Hiro marker a protein Table 1 Links to all web examples arranged by figure. See https://lucianoabriata.altervista.org/papersdata/tablepeerjcs2019.html for an online version of the table which also includes links to several video demonstrations. Figure 2 2A https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/2brbutane.html 2B and 2C https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/pdb-1vyq-1fjl.html 2D https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/bacteriophage.html 2E https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/xray3hyd.html 2F https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/bh3nh3orb.html 2G https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitinandNESatomistic.html 2H https://lucianoabriata.altervista.org/jsinscience/arjs/jsartoolkit5/pdbloader6.html Figure 3 3A https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmolclashdetection.html and https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmolprotontransfer.html 3B https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmoldielsalder.html 3C https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/metmyoglobinfe3pcshift.html Figure 4 4A https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitinuimffvoicesaxs.html 4B https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/coevol_1qop.html Figure 5 5A–5C https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitin-uim-cannon.html Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 8/23 http://dx.doi.org/10.7717/peerj-cs.260/supp-1 http://dx.doi.org/10.7717/peerj-cs.260/supp-1 https://lucianoabriata.altervista.org/papersdata/tablepeerjcs2019.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/2brbutane.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/pdb-1vyq-1fjl.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/bacteriophage.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/xray3hyd.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/bh3nh3orb.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitinandNESatomistic.html https://lucianoabriata.altervista.org/jsinscience/arjs/jsartoolkit5/pdbloader6.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmolclashdetection.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmolprotontransfer.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/smallmoldielsalder.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/metmyoglobinfe3pcshift.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitinuimffvoicesaxs.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/coevol_1qop.html https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ubiquitin-uim-cannon.html http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ complex rendered as cartoons with small molecule ligands rendered as sticks (PDB ID 1VYQ, same example used by Berry & Board (2014)); and Fig. 3 shows a cartoon representation of a protein bound to a short segment of double stranded DNA rendered as sticks (from PDB ID 1FJL) spinning on the Kanji marker. Figure 2D exemplifies display of VMD isosurfaces to show a volumetric map of a bacteriophage attached to its host as determined through electron microscopy (EMDB ID 9010). Two further examples feature combined representations of atomic structure and volumetric data: Fig. 2E shows a small peptide rendered as sticks and docked inside the experimental electron map shown as a mesh (from PDB ID 3HYD), and Fig. 2F shows the frontier molecular orbitals of BH3 and NH3 (from Wavefront files kindly provided by G. Frattini and Prof. D. Moreno, IQUIR, Argentina). The next level of complexity is building up scenes from 3D primitives, which brings the advantage over ready-made models that all the elements can be handled independently, thus allowing the possibility of incorporating interactivity. This can be achieved either through A-Frame with AR.js, thus requiring only HTML code for display as in the Wavefront-based examples above, or through libraries that require additional JavaScript code but in turn enable more complex functionalities. Exemplifying the former case, Table 2 Requirements and troubleshooting for developers and users. Developers Software and hardware requirements * Ensure using https URLs; otherwise webcams will not activate. * Free web hosting services work, as web pages only need to be hosted but run in the client. * Given the regular updates in w3c standards, APIs and libraries, routine tests are recommended to ensure normal functioning. * Examples from this paper were verified to work “out of the box” on Safari in multiple MacOS 10.x versions and on multiple Chrome and Firefox versions in Windows 8, Windows 10, Linux RedHat Enterprise Edition, Ubuntu and ArchLinux. * Currently limited and heterogeneous support in tablets and smartphones, these devices are not recommended. Users Software and hardware requirements * Need a webcam- and internet-enabled computer (desktop or laptop). * Enable webcam when prompted. * JavaScript and WebGL must be enabled in the browser (that is a default setting). * Check that ad blockers and firewalls do not block the webcam and other content. * Pages containing large Wavefront files may take time to load (half to a few minutes). Augmented reality markers * Print on regular printer; try different sizes for different applications. * When using Hiro and Kanji markers ensure they are printed at the same size. * Ensure that makers have a white frame around the black drawing, at least 10% of size. * To improve marker recognition, avoid glossy papers. Opaque paper is best. * Lighting may also affect the quality of the AR experience. * Markers are easier to handle if glued on solid surfaces (but avoid wrinkles). * Cubic marker can be glued on solid rubber cube cut to appropriate size. Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 9/23 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ Fig. 2G uses A-Frame spheres and cylinders placed at coordinates computed from atomic positions, colored by atom type and assigned atom-specific radii, to render a model of a small protein helix on the Kanji marker. The latter case of using pure JavaScript is Figure 2 Different implementations of WebGL for AR in web browsers. (A) 2-bromo-butane enantiomers R and S displayed on Hiro and Kanji markers, respectively, from Wavefront objects pro- duced in VMD. (B) A protein complex rendered as cartoons with small molecule ligands shown as sticks on the Hiro marker (PDB ID 1VYQ). (C) A double-stranded segment of DNA (sticks) bound to a homeodomain protein (cartoon) spinning on the Kanji marker (PDB ID 1FJL). Both B and C were rendered from VMD Wavefront objects. (D) Volumetric map of a bacteriophage attached to a portion of the host’s cell wall as determined through electron microscopy (EMDB ID 9010) and prepared as VMD Wavefront object showing the capsid in gray, the needle in blue and the membranes in orange. (E) A small peptide inside its electron density map as determined by X-ray diffraction (PDB ID 3HYD). (F) HOMO and LUMO orbitals from NH3 and BH3 molecules, rendered from Wavefront objects. (G) Representation of an amphipathic alpha helix built from primitives, viewable on the Kanji marker. (H) Use of a cube marker (made up of six different AR markers in its faces) to load any molecule in PDB format and handle and visualize it in 3D. Graphics built from Three.js primitives. The example also uses Cannon.js to simulate rigid body dynamics by fixing the distances between atoms separated by one or two bonds but allowing rotations, in analogy to plastic-made molecular modeling kits. Full-size DOI: 10.7717/peerj-cs.260/fig-2 Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 10/23 http://dx.doi.org/10.7717/peerj-cs.260/fig-2 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ illustrated by the jsartoolkit library (https://github.com/artoolkit/jsartoolkit5), in an example where six markers arranged on the faces of a cube are coupled to drive a single object in full 3D. Lastly, combining this with direct use of Three.js to draw 3D primitives, the web app in Fig. 2H allows visualization and manipulation of any molecule loaded in PDB format in full 3D, in this case (D)-glucopyranose. Adding interactivity: examples on small molecules Web apps using A-Frame can gain interactivity through portions of JavaScript code that read atom coordinates and carry out calculations on them. Figure 3 shows another series of examples of increasing complexity, focusing on small molecules. In Fig. 3A, the user drives a lysine side chain with the Hiro marker and a glutamate side chain with the Kanji marker. Each molecule is anchored to the center of its corresponding AR marker through its alpha carbon atom. Their protonation states correspond to neutral pH, so lysine is protonated hence its N atom (blue) is charged by +1, whereas glutamate is deprotonated hence its O atoms (red) bear a total charge of −1. Through simple JavaScript code the web app (i) represents with yellow dots the vector connecting the lysine’s N atom and one of the glutamate’s O atoms; (ii) reports the distance between these atoms and the corresponding attractive electrostatic force in real time; and (iii) checks for and displays clashes between any pair of atoms of the two molecules. The code for (i) is wrapped inside Figure 3 Introducing interactivity. (A) A lysine and a glutamate side chain attached to different AR markers, whose coordinates are processed in real time to deliver distance and electrostatic potential between charged groups and to calculate and display clashes. Graphics achieved with A-Frame polyfills. In this example the ratio of protonated lysine to glutamate exchanging dynamically is set to 70/30 = 2.33, that is, favoring protonated lysine as the actual water chemistry dictates but shifted orders of magnitude from the real ratio of acidic constants so that the user can observe temporal protonation of both amino acids in the timescale of work. (B) A Diels-Alder reaction, taking place once the user has moved the reagents close enough in space; this example further helps to visualize fused cycles (as the diene reagent was a cyclic molecule itself). (C) Example of interactively probing experimental observables: as a probe atom (black sphere) is moved around a paramagnetic center, the user sees the paramagnetic effects simulated at the location of the probe atom in real time. Full-size DOI: 10.7717/peerj-cs.260/fig-3 Abriata (2020), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.260 11/23 https://github.com/artoolkit/jsartoolkit5 http://dx.doi.org/10.7717/peerj-cs.260/fig-3 http://dx.doi.org/10.7717/peerj-cs.260 https://peerj.com/computer-science/ an auxiliary .js file, and the code for (ii) and (iii) is located between