key: cord-0674091-oybljqhg authors: Pournaras, Evangelos; Ghulam, Atif Nabi; Kunz, Renato; Hanggli, Regula title: Crowd Sensing and Living Lab Outdoor Experimentation Made Easy date: 2021-07-08 journal: nan DOI: nan sha: b73589eba169b487b17ce491901e7b5854b80342 doc_id: 674091 cord_uid: oybljqhg Living lab outdoor experimentation using pervasive computing provides new opportunities: higher realism, external validity and socio-spatio-temporal observations in large scale. However, experimentation `in the wild' is complex and costly. Noise, biases, privacy concerns, compliance with standards of ethical review boards, remote moderation, control of experimental conditions and equipment perplex the collection of high-quality data for causal inference. This article introduces Smart Agora, a novel open-source software platform for rigorous systematic outdoor experimentation. Without writing a single line of code, highly complex experimental scenarios are visually designed and automatically deployed to smart phones. Novel geolocated survey and sensor data are collected subject of participants verifying desired experimental conditions, for instance, their localization at certain urban spots. This new approach drastically improves the quality and purposefulness of crowd sensing, tailored to conditions that confirm/reject hypotheses. The features that support this innovative functionality and the broad spectrum of its applicability are demonstrated. IS THE FUTURE of scientific human experimentation outdoors? Pervasive computing shapes this future on the basis of a large market for smart phone hardware and software. Ubiquitous devices turn into living laboratory instruments awaiting to directly collect real-world observations at massive scale to tackle some of the most challenging research questions and hypotheses [1] , [2] , [3] . In this future of Smart City computing applications such as transportation, mobility and sharing economies, new epistemological and methodological challenges [4] arise for social and environmental science, economics or psychology. For instance, how to practice causal inference, how to mitigate trade-offs of realism vs. environment controllability, how to handle biases and counterfactuals [5] ? The COVID-19 pandemic makes these questions timely and pressing given the safety measures against infection risk in indoor laboratory environments. Are existing online approaches [5] cost-effective for outdoor ubiquitous experimentation? Experimentation 'in the wild'? Not easy Several existing efforts focus on online web experimentation. For instance, nodeGame is a Javascript-based tool for online real-time synchronous decision games and experiments [6] . Similarly, z-Tree is a client-server software for ready-made economic experiments [7] and oTree is a Python framework that supports multiplayer or single-player strategy games, dynamic questionnaires and assessments/tests [8] . Experimentation based on pervasive computing includes AWARE, an open-source mobile instrumentation toolkit that focuses on the resource constraints of mobile devices [9] . StimuliApp is designed to create psychophysical tests with precise timing on iOS and iPad devices [10] . A simple system of menus can support the design and running of tests that assess perceptual capabilities. Another way to collect geolocated crowd sensing data is the so called 'digital public notice areas', however, their use requires specialized software/hardware and they mainly support commercial applications [11] . The AIRBOT mobile information assistant is proposed for complex tasks in public space. It has been used to support air passengers in successfully boarding on a plane [12] . Several limitations are identified in these existing approaches: (i) The design of an experiment can be complex and usually requires technical/programming skills to achieve rigor or implement a more systematic collection of highquality data. (ii) A seamless integration to realworld outdoor activities is usually limited. (iii) No general platform approach to outdoor experimentation -pervasive system tools are limited to specific experimental and application scenarios. This article addresses these challenges by introducing the solution of Smart Agora (https:// smart-agora.org). Designers visually create complex crowd sensing experimental scenarios on a web browser and deploy them to run without moderation to the smart phones of participants. The collected data are of high quality and sociospatio-temporal in nature, purposeful to minimize privacy cost, noise and biases. The living lab experimental processes of Smart Agora are tailored by design to satisfy desired experimental conditions, for instance, geolocated survey responses verified by location proofs of participants at certain urban spots [13] , e.g. on the blockchain. The broad applicability of Smart Agora is found at several Smart City domains such as cycling risk, urban infrastructures, decisions for participatory budgeting projects and other [13] , [14] , [15] . The contributions of this article are summarized as follows: • A flexible and modular modeling architecture for crowd sensing and outdoor living lab experimentation that supports a large spectrum of complex experimental scenarios. • The open-source platform of Smart Agora that realizes the modeling architecture by implementing the Smart Agora Dashboard for designers of experiments and the Smart Agora App for recruited participants. • A review of Smart Agora use cases to demonstrate the broad spectrum of its applicability. • A software artifact demonstrator [16] of Smart Agora running on a virtual machine for reproducability, engagement and assessment of the approach by the broader community. • Systematic documentation of Smart Agora for users and developers. The 'Software Accessibility' is outlined at the end of this article. The Smart Agora living lab is an open-source crowd sensing framework for visually designing and carrying out complex outdoor socio-spatiotemporal experiments. It consists of the Smart Agora Dashboard and the Smart Agora App. The Smart Agora platform is documented and can be easily tested in a provided virtual machine [16] . All these are listed in 'Software Accessibility', at the end of this article. The Smart Agora Dashboard is a web platform designed to simplify the creation of complex outdoor experimental scenarios using pervasive devices such as smart phones. The designer of Smart Agora experiments creates geographical points of interest where participants share experimental data, e.g. geolocated survey and sensor data. This dashboard provides the option to reuse and modularize several complex data collection processes as well as deploy them at different time points with different participants. The design of experimental scenarios is visual and interactive without the need to write a single line of code. However, each experimental scenario can be fully encoded and parsed within JSON files that allow automated generation of complex experiments. For instance, consider experiments that require the collection of data at every bus or tram station in a city. The points of interests can be automatically loaded by including the location of the stations in the JSON files, along with the necessary data collection configurations. The following JSON 1 code listing illustrates an example of how data collection is encoded. JSON 1: Example of a data collection asset. It contains configurations for a point of interest coming with a survey question with two options to choose from as well as two sensor data collections (gyroscope and GPS location). "Id": "AXeG00HIQMa8aD8nfimV", 3 "Name": "Simple_09022021_134527", 4 "Url": "http://smart-agora.org", 5 "Metadata": { 6 "record": { 7 "StartAndDestinationModel": { 8 "StartLatitude": null, 9 "StartLongitude": null, 10 "DestinationLatitude": null, 11 "DestinationLongitude": null, 12 "Mode": "Simple", 13 "DefaultCredit": "3" 14 }, 15 "SampleDataModel": [{ 16 "id": 1, 17 "Question": "How dangerous for bikers was the last section? ", 18 "Type": "radio", 19 "Latitude": "47.3715915", 20 "Longitude": "8.538603799999999", 21 "Sensor": [{ 22 "id": 1, 23 "Name": "Gyroscope" 24 },{ 25 "id": 2, "Name": "Location" 27 }], 28 "Time": "3", 29 "Frequency": "Medium", 30 "Sequence": "Disable", 31 "Visibility": "true", 32 "Mandatory": "true", 33 "Option": [{ 34 "id": 1, 35 "Name": "Safe", 36 "NextQuestion": null, 37 "Credits": "" 38 },{ 39 "id": 2, 40 "Name": "Dangerous", 41 "NextQuestion": null, 42 "Credits": "" 43 },], 44 "Combination": null, 45 "Vicinity": "25" The Smart Agora app is an Android app that allows participants to subscribe and participate to experiments. These experiments are loaded in an augmented map that contains the points of interest that participants visit to answer questions on spot, while sharing smart phone sensor data (see Figure 1 ). Visiting the points of interests is verified when participants are in close proximity with a point of interest measured by a radarlike radius of a localization cycle. A similar method with an ellipse is also supported, where the shortest axis of the ellipse points to the direction of the next point of interest [17] . This elliptical method of localization could be suitable in urban environments with narrow streets. Smart Agora localization currently operates based on GPS technology. However, approaches based on distributed ledgers operating with LoRaWAN and adhoc networks can be more secure and reliable alternatives [13] . Smart Agora supports by design innovative experiments that improve data collection quality. Compared to continuous privacy-intrusive sensing approaches [18] , Smart Agora collects minimal data, tailored to specific context and purpose and as such it is easier for the designers of the experiments to meet minimal data collection requirements of ethical review boards. This is made possible by making participants' responses subject of proving witnessed presence [13] . Answering survey questions and sharing sensor data is performed after verification of the location or another criterion related to the situational awareness of participants using QR codes, challenge questions, puzzles and CAPTCHA-like tests. This is useful in experiments with outdoor group interactions in which group members require to prove their identity and membership to each other group member. Similarly, the observations of experiments involving physical objects, e.g. the effect of a political poster, the response of a traffic light for pedestrians or arrival delays in a bus station, could be verified with a QR code on the physical object. These verification processes result in automating the control of experimental conditions that is usually performed manually by lab moderators in indoor lab environments. Proving witnessed presence has a significant impact on improving the quality of data collection. Data sharing is designed as responsible testimonies and citizens' interventions [13] . This makes Smart Agora highly versatile and applicable in different living lab manifestations: (i) Science -scientists design experiments and participants are recruited to participate. Cycling risk assessments is an example here [14] , [13] . (ii) Policy-making -policy makers design participatory decision-making and deliberation processes, while citizens participate directly in crowd-sourced decisions and bottom-up policy formation. The participatory budgeting field test in the city of Aarau is an example here [15] . (iii) Education -teachers and lecturers design highly engaging outdoor learning activities and students learn out of the class. The 2018 MSc course "Data Science in Techno-socio-economic Systems" at ETH Zurich is an example here. Figure 2 introduces the modeling architecture of Smart Agora. It consists of four core elements: (i) project, (ii) asset, (iii) task and (iv) assignment. A crowd sensing project can contain several data collection assets that encode experimental scenarios. Each asset can be used in several unique data collection processes by creating a task and linking this task with the asset via an assignment to participants. A task results in data collected by the assigned participants according to an asset. All experimental processes can be designed visually without writing a single line of code. The Smart Agora architecture is implemented by the hive platform built in Go and Elasticsearch. Figure 3 illustrates the lifecycle of a Smart Agora experiment from design to deployment and completion of the data collection. The designer of the experiment creates an account to the Smart Agora Dashboard and logins to create a crowd sensing project. The participants of an experiment download the Smart Agora App and are granted access to the subscription of projects using an access code provided to them by the moderators of the experiment as part of the recruitment process. In the meantime, the designer of the experiment creates data collection assets that contain the points of interests with the survey questions and the desired mobile sensor data for collection. These assets are associated with tasks via the creation of assignments that control which participants can join the experiment. The participants can subscribe to the task assignments. This loads the points of interests and survey questions to their smart phone map, while it configures the sensor data collection, e.g. frequency. Upon completion of the task, the data are transferred to the Smart Agora database for storage and analysis. Smart Agora also provides the option to forward collected data to DIAS (http://dias-net.org), the Dynamic Intelligent Aggregation Service [19] for decentralized real-time data analytics [13] . The rest of this section illustrates the features of Smart Agora and how they support different outdoor experimental scenarios. A Smart Agora crowd sensing project is the top of the modeling hierarchy and facilitates a group of experimental processes and collected data. It is initially created by the designer of the experiment and contains multiple crowd sensing assets, tasks and assignments. By default, an auto-assignment option connects a created asset with the most recent task and is made available immediately to all available participants in the pool. A crowd sensing project can be used to group together a series of several experiments and collected data that serve a research project. In this way, the whole experimental design of a research project can be built incrementally and managed more efficiently. The crowd sensing asset provides toolkits to design ubiquitous experiments and the collection of socio-spatio-temporal data. An asset is associated with a project. An experiment can be conducted in three navigational modalities: • Simple modality: Navigation to different points of interests occurs in any order. This experimental modality is provided for experiments with a large number of points of interests (e.g. all bus stations of a city), probably expanding to a large geographic area of several possible participants. Data collection is usually long-term and large in scale, making it suitable to compare experimental observations over different geographical areas and over time, i.e. evolving human behavior and environment. • Sequential modality: Navigation to different points of interests occurs in a predetermined order introducing in this way a well structured experimental process that has to be followed by participants (e.g. following given vehicle routing trips). The information into which a participant is exposed at a certain point of interest determines the behavior and survey responses at the next points of interests. This allows to observe and study systematically a socio-spatiotemporal phenomenon (e.g. mobility pattern) by tracing at check points (the points of interests) how it incrementally evolves. Compared to the simple navigation modality, this one is suitable for more controlled experimental scenarios with a lower number of points of interests and participants, with the geographical scale depending on the transport mode involved, e.g. walking pedestrians vs. cyclists, vs. vehicle drivers. solving. An experiment conducted under the dynamic modality can be used to vary onthe-fly experimental conditions of participants based on their demographics or environmental variables they experience and detected within the sensor data. Moreover, an asset consists of multiple geolocated survey questions. Each question consists of a location on a map (GPS coordinates), the question formulation, the question type (radio, checkbox, likert scale and text box) and an optional amount of rewards that will be provided to the participants for answering this question. These rewards can represent the recruitment compensation and incentives to preserve engagement or scale up data collection, while minimizing attrition effects [5] . Rewards can be monetary, discount coupons or capture value measured with a designed cryptocurrency (token). Smart Agora can couple with other systems to automatically manage transactions and the actual secure value exchange, for instance digital wallets that verify transactions on distributed ledgers (blockchain). Depending the question type, the designer may introduce the options to choose from. Additional rewards may be provided for each selected option to allow experimental economics research. Each point of interest can be further augmented with the collection of smart phone sensor data including sensors such as: light, gyroscope, proximity, accelerometer, GPS location and noise. Sensor data are collected in a determined vicinity around the point of interest during a determined duration and at a low (every 2 sec), medium (every 250 ms) or high (every 200 ms) frequency. A large localization radius can provide unrestricted collection of sensor data in experimentation scenarios requiring a continuous and passive crowd sensing data. Sensor data and sensor fusion can be used for human activity recognition (traveling vs. walking vs. cycling) as well as social inferences (who speaks with whom on spot), including the inferences to prove witnessed presence [13] . The creation of an asset is a collection of multiple points of interests configured to collect any combination of survey question data and/or smart phone sensor data. They are created interactively via a visual wizard that guides the designer step by step to build a crowd sensing asset. At the end, this wizardl generates files, such as the JSON 1 code, that represent the experiment and the designer's choices. Several assets can represent different control and treatment groups for comparisons. A crowd sensing task is determined within a project. It is associated with an asset via an assignment and as such it contains the collected experimental data. Tasks allow a degree of freedom in the data that can be collected by (re)using one or more assets. For instance, given a crowd sensing asset, the same data can be collected at different time points by the same or different groups of participants. And vice versa, the same data can be collected by different groups of participants at the same or different time points. An assignment connects an asset and a task to the participants of an experiment. The designer has control of which participants to choose from the ones recruited and available in the pool. Four different use cases are outlined here to demonstrate the applicability spectrum of Smart Agora and its impact on outdoor experimentation. The capability of Smart Agora to facilitate informed decisions by geolocating the decisionmaking process (proof of witnessed presence) is assessed in a study on cycling safety [13] . The perception of bike riders about the cycling safety of different urban spots in Zurich is compared to an empirical safety model [14] built using official data of the Federal Roads Office collected from Swiss GeoAdmin. If the two safety estimations match, this is indication that witnessed presence in participatory crowd sensing can indeed provide information quality comparable to the official but costly data collection methods. The empirical risk estimation of a route is derived by a continuous spatial accident risk model of Zurich [14] . This is the baseline that is compared to the perceived cycling risk estimated via 11 Smart Agora test users. A crowd sensing asset is designed using the sequential navigation modality with four points of interest. The cycling risk of each road section that connects two consecutive points of interests is assessed at the latter point. A likert scale question pops up in the Smart Agora app to evaluate cycling risk at a linear scale between 1. very safe to 5. very dangerous. All test users cycled at the same day and time to minimize biases originated from weather, light condition and the condition of different bikes. A high matching is found between the empirical evidence from the cycling accident data and the cycling risk that citizens witness [13] . This motivates an alternative more legitimate policy-making approach that relies on highquality crowd sensing data collected by verifying witnessed presence. Along with collecting high-quality policymaking data, Smart Agora can also empower novel participatory processes that have earlier been too costly or complex for digital democracy. For instance, a participatory budgeting field test is planned to run in the city of Aarau in Switzerland, where citizens will decide themselves for a Smart City project to fund using a municipal budget of 10K CHF to improve their life [15] . Examples of such projects include transport decarbonization initiatives (e.g. bike sharing stations), digital cultural heritage, smart waste bins and other. The use of Smart Agora for outdoor experimentation opens up several new research avenues that will be explored in this project. For instance, to what extent proving witnessed presence results in more legitimate voting outcomes? Letting citizens of a local district community to witness urban spots that will be transformed by Smart City projects has an added value at different levels: Citizens can physically come closer to each other to deliberate about the impact of the project on the community and urban surrounding. Citizens can improve their awareness by collecting evidence to support a more informed and responsible decision. This process reunites the digital and physical decision space of citizens, inspiring the revive of a new cyber-physical 'ancient Agora of Athens'. Ultimately, this process shall result in a voting outcome accepted by the broader community and even by citizens that voted against the elected project [15] . Moreover, voting rules, e.g. majority voting vs. multi-option preferential voting, can influence voting outcomes as well as the capacity of local communities to reach legitimate decisions that promote consensus. Smart Agora can implement different voting rules applied to different treatment groups during the field test to make novel comparisons in the context of outdoor geolocated decision-making. Beyond the persistence of collected data in the Smart Agora database for offline analysis, the integration of DIAS to Smart Agora provides capabilities for decentralized real-time data analytics [13] . In particular, collective measurements (summation, average, max, min, count, etc.) can be made and updated in real-time based on the most recent shared data of participants localized at points of interest. When participants depart from a point of interest, automated rollback operations are performed to preserve accurate estimations using exclusively localized participants. These novel calculations are performed in an interactive and cooperative fashion by participants' devices and without the involvement of any centralized third party. DIAS implements the distributed communication protocol and a memory system for efficient localized data management, see earlier work for technical details [19] , [13] . To demonstrate this progressive capability, a complex field test was conducted in Zurich with 6 test users to assess the sustainability of transport usage [13] . The participants choose the preferred transport mean to reach two points of interest using a simplified linear model of sustainability: 0. Car, 1. Bus, 2. Train, 3. Tram, 4. Bike, 5. Walking. The mean sustainability is updated in real-time as participants arrive and depart from the points of interest, where they choose their preferred transport mean, see the video at 'Software Accessibility'. The observed estimations of transport sustainability approximate well the actual sustainability values. This confirms the efficiency of Smart Agora to carry out decentralized realtime collective measurements. The COVID-19 pandemic creates several challenges for indoor experimentation. Participants and moderators are exposed to significant infection risks. Several university labs have been closed during lockdowns, while setting up a safe lab can be costly, complex and time-consuming, requiring institutional approvals along with new frameworks of safety and operation (ventilation, disinfection, COVID-19 testing, recruitment, ethical approvals, etc.). In contrast, outdoor experimentation with Smart Agora reduces infection risk for partici-pants as experimental tasks are deployed automatically and remotely. Smart Agora can also provide a higher degree of freedom to scale up recruitment and ubiquitous data collection, lower moderation cost and higher realism. Moving experimentation outdoors provides new opportunities to limit sampling biases of indoor lab pools of participants, often overpopulated by young individuals with a student profile. Participants from older age groups can be engaged given the priority they have to vaccination as well as the opportunity to eliminate isolation of elderly during lockdown and to contribute to public good. Recruitment for experimentation can be heavily affected due to COVID-19 governmental constraints on mobility. These can restrict people to move only in close proximity for essential purposes such as going to work, shopping or exercising. Other activities may be prohibited and even fined. Smart Agora provides new opportunities to seamlessly integrate experimental processes within permitted mobility ranges. For instance, instead of an indoor driving simulator, driving behavior can be studied during pandemics when citizens drive to work or for shopping by implementing a travel diary using Smart Agora. Similarly, Smart Agora can also serve COVID-19 research [3] , [2] , for instance understanding infection risk and epidemic spread by crowd sensing collective human behavior and actions. Incentives to participate in experimentation may deviate from the norm during pandemics. Lab recruitment can be a flexible income support given the loss of jobs. Non-monetary incentives have a potential to strengthen the sense of community belonging using participatory and gamification elements [20] such as the ones of Smart Agora. These incentives may capture new ways to trigger intrinsic human values during lockdowns, for instance, higher interest in contributing to science, i.e. citizen science, outdoor exercising or well being. As Smart Cities co-evolve to complex ecosystems of compelling socio-technical complexity, living lab outdoor experimentation will remain a timely and significant challenge to tackle. Collecting crowd sensing data is not only the end goal but also the mean to reason about desired exper-imental conditions to carry out rigorous studies involving causal inferences. Verifying such conditions in an automated way opens up new avenues to carry out large-scale outdoor experimentation with lower biases, noise and moderation level. Smart Agora is an endeavor along these lines, made to support both experiment designers and participants in different manifestations of science, policy-making or education. Several challenges though remain such as how to incentivize participation and minimize dropouts as well as identify and minimize unknown biases that will emerge in complex urban environments. Outdoor experimental participation requires new intrinsic values for incentivizing citizens' contributions to science. Distributed ledgers and cryptoeconomic principles such as token curated registries are promising means to capture and securely exchange such values. This is also expected to scale up the number of participants in future use cases. Despite the minimal and targeted data collection approach of Smart Agora, solutions for privacy protection and data sharing control will be implemented in future, for instance, differential privacy and homomorphic encryption to privately aggregate sensitive citizens' data. Participants may require self-determined access control rights to sensor data to deliver certain crowd sensing tasks [19] . Hence, data collectors need to match with data suppliers based on shared values that go beyond monetary ones and resolve social dilemmas related to privacy, ethics, quality of life, comfort and other. For instance, citizens' assemblies may play a role of ethical review boards determining experimental participation and as such paving the way towards new standards and norms on how to conduct experimental research. This will allow citizens to 'shape' and ethically align experiments to their own common values and in a bottom-up way, pushing towards a more socially responsible science. The Smart Agora platfrom and supporting material are open-source and accessible via the links below: Prof. R. Hänggli is full professor of political communication at the University of Fribourg (Switzerland). From 2011-2013 she was assistant professor at University of Amsterdam. Before, she studied Political Science at the University of Berne, and got a PhD in 2010 from the University of Zurich. Her work deals with (digital) democracy and opinion formation processes, and the interplay between politicians, media, and citizens. She is a Principal Investigator in the NRP 77 "Digital Transformation" with a project about digital democracy. • The hive platform on which Smart Agora relies A systematic review of living lab literature Mobile phone data for informing public health actions across the covid-19 pandemic life cycle Give more data, awareness and control to individual citizens, and they will help covid-19 containment Bit by bit: Social research in the digital age Conducting interactive experiments online nodegame: Real-time, synchronous, online experiments in the browser z-tree: Zurich toolbox for ready-made economic experiments otree-an open-source platform for laboratory, online, and field experiments Aware: mobile context instrumentation framework Stimuliapp: psychophysical tests on mobile devices Digifieds: insights into deploying digital public notice areas in the wild Towards task-sensitive assistance in public spaces Proof of witness presence: blockchain consensus for augmented democracy in smart cities On cycling risk and discomfort: urban safety mapping and bike route recommendations Humancentered democratic innovations with digital and participatory elements Annual International Conference on Digital Government Research Smart agora software artifact demonstrator Sensing and mining urban qualities in smart cities Engineering democratization in internet of things data analytics Gamification in theory and action: A survey The authors would also like to thank Prof. Dirk Helbing and the Chair of Computational Social Science for supporting this research. Edward Gaere supported the integration of Smart Agora with DIAS [19] , while Dr. Alexey Gokhberg provided the Hive interfaces to Smart Agora. The authors would also like to thank the 2018 students of the course "Data Science in Techno-socio-economic Systems" at ETH Zurich who used Smart Agora and provided invaluable feedback. Many thanks go to other Empower Polis team members as well as the Institute of Science Technology and Policy (ISTP) of ETH Zurich for running the ETH Policy Challenge and providing a venue to cultivate ideas for outdoor living lab experimentation. ACKNOWLEDGMENT This research is funded by the SNF NRP77 'Digital Transformation' project "Digital Democracy: Innovations in Decision-making Processes", #407740 187249:https://www.nfp77.ch/en/portfolio/ trust-and-legitimation-in-the-digital-democracy/.