Capturing Emotional Reactions to Data Visualization

Background

Data visualizations, especially those with interactive components, are a recent hot topic in the intersection of tech and journalism. They are an engaging way of presenting volumes of information but incur challenges related to the assessment of the effectiveness of these visualizations. State-of-the-art technologies that can use live video footage to capture and analyze people’s emotional responses are one way of deciphering reactions to data visualizations. However, technical and design challenges will still need to be addressed with respect to linking interactive visualizations with emotion-capture technologies, keeping records of real-time emotional states, and graphically rendering people’s interactions with these visualizations to shed light on how they can improve.

Executive Summary

Sets of visualizations already exist and are plentiful on news organization websites, for example. The primary task is in developing a general tool that can extract emotional state/interaction data from any person’s experience with a visualization. We would need to look into pre-existing libraries that integrate with webcams to analyze and detect changes in emotional state in real-time. It may be helpful to construct a platform that amasses “emotional profiles” of users and detects patterns or irregularity in their use of a range of visualizations and stimuli. Stripping identifying information from these profiles as well as encrypting logs is essential when implementing this platform. To ensure the accuracy of the computed emotional reactions, a post-quiz should be given to participants who can indirectly validate or invalidate the results based on their own perceptions of levels of confusion, fascination, or interest in the visualizations they were presented with.

Viability Analysis

Finding open-source or non-licensed libraries for emotion-capturing webcam software may be difficult, depending on technology patents. Deciding the best way to represent extensive logs of emotional states in aggregate will require some research insofar as choosing a best visualization as well as safeguarding the privacy of participants. Consistency in factors such as background lighting and webcam quality may prove to be very important, so it may be best to administer the interactions in a controlled environment on Bucknell computers where we can control as many variables as possible — but this may become challenging if the client hopes to make use of the tool by recruiting participants around the world (on Amazon’s Mechanical Turk, for instance).

Risks and Rewards

Risks include privacy violations in light of IRB restrictions as well as falsely representing the emotional reactions of participants which could go on to influence decisions of data visualization creators and designers. Rewards include potential to interact with visualizations in new ways by garnering a better understanding of what works and what doesn’t across demographic categories and different people. Such improvements to data visualizations as a result of the feedback provided by this tool will positively impact active learning and engagement with an array of data, ranging from politics to science or any sort of quantitative observation.

Closing

The development of this system will be an instrumental stepping stone if we hope to continue disseminating information effectively and pragmatically on the web. A voluntary or crowdsourced platform that assesses emotional impact while protecting the privacy of its users has enormous potential to positively reinforce quality visualizations and critique poor visualizations, to the effect of creating and encouraging better, more informational online resources.

Project Graverobber

Background

A pre-existing system needs to be either revisited or recreated from scratch in order to remain viable for regular use by the parish cemeteries in Shamokin, PA. The problem being addressed is the cataloguing of graves in the local area, an undertaking that is imperative to the daily operations of the cemeteries. Since the original developer passed away a year ago, staff has been unable to continue using the program and so usability is likely a top concern. Organization of data and research are also priorities when tackling this project.

Executive Summary

A more modern alternative to the computer program and physical CDs would encompass a web and/or mobile application with cloud storage that syncs recent burials across all local cemeteries. A web/mobile app designed with user friendliness in mind would allow the parish secretary to seamlessly update the records of graves whenever new ones are added (more research would need to be conducted into the cemetery’s daily operations and how exactly new entries are handled, electronically or otherwise). Other administrative features that could benefit the church, such as relational database features that alert parish staff when older graves need maintenance, may also be considered depending on interest.

Viability Analysis

The creation of a simple, intuitive and secure system that handles the cataloguing and insertion of new graves seems well-suited to a full-stack consisting of a UI (for sorting, adding, and viewing details of graves) and an underlying database (that’s responsible for securely storing the details of all burials in a given system of cemeteries). Challenges will include deciding on the best course of action (web, mobile, or both) and the best technologies for creating the stack (front-end frameworks, server host, database selection, etc.)

Risks and Rewards

The completion of this project would yield a helpful resource for parish staff. A generalizable tool beyond Shamokin cemeteries could impact a number of administrative collections of entries, not just burials.

Closing

The creation of a sleek and user-friendly application (perhaps more generalizable than a catalogue for Shamokin cemeteries specifically) would be an interesting and valuable project. The ability to seamlessly add new information while sorting and browsing the old, via a transition to cloud storage rather than antiquated CDs, would be a good starting point in terms of goals for this project. Later iterations could include additional functionality to assist in the administrative efforts of the staff.

Film Findr

Background

The client wants to improve upon an existing database of feature films for use by scholars and film buffs. A preliminary database has already been implemented, but additional improvements to the film upload process, search features, and user interface are desired. The proposed system is already live on a Bucknell server and would benefit film & media studies students and faculty in addition to anyone else with internet access who was interested in the scope of feature-length films.

Executive Summary

Solutions to the three identified problems: 1) Film upload process — Web crawlers and regular visits to the “recently added” portion of the IMDB site would provide autonomous clips, screenshots and metadata from a large variety of feature-length films. 2) Search engine — More robust search features (there are many possibilities; e.g. searching by actors, by scenes, subtitles in foreign languages, type of shot, color [would require image processing of screenshots], and more). Use metadata extensively to streamline searches and cut down on raw video footage being uploaded to server (very expensive to host on AWS for long periods of time). Machine learning would be another interesting venue to explore in this regard (could pick up on frequently recurring colors, motifs, expressions, shots, actors, etc.) OR crowdsourcing this to an extent would be another possibility and facilitate the growth of an online community. 3) User interface — Would need to see the current UI or hear about specific problem areas to brainstorm ideas at this stage. Intuitive menus, search bars and compact/aesthetically pleasing interfaces can easily be achieved with Bootstrap and elegant animations/scripting functionality can be added with JavaScript as needed.

Viability Analysis

One constraint is the client’s use of AWS, a cloud service that can charge a lot of money for hosting files as large as a database of movie clips (or even screenshots and/or subtitles). An LRU (least-recently-used) principle may help mitigate this constraint, where movies that have not been queried for at least 6 months can be removed from the live site and free up space for more popular titles. Copyright violations should also be closely considered — an educational database for film students is likely fair use, but we would need to more closely research the implications of uploading relatively long clips, hundreds of contiguous screenshots, or entire movie scripts. The prospect of automatically uploading film data is another difficult obstacle and would necessitate one of several solutions, one of which is machine learning and would likely take more than one semester to procure satisfying, consistent results.

Risks and Rewards

Aside from technical experience and exposure, updating a Bucknell film database would provide a more intuitive and streamlined resource for film students and faculty who want insight into film quotes, footage and more. Undertaking an overhaul of the film upload process would be ambitious and risky, but also extremely rewarding if accomplished (would save users a lot of time and vastly improve the selection of films to include more obscure or indie titles).

Closing

Improvements to the film upload process, search functionality and UI would be beneficial to the continued updates of Bucknell’s film database. There are several avenues that would increase the quality of users’ experience while using the site and introduce them to less common films while providing them with accurate information.

Link to PDF Version

Brooke Bullek

WORK EXPERIENCE

Software Developer (Front-End) – Department of Chemical Engineering, Bucknell University, June 2017 – Present

• Designed and implemented a set of chemistry simulations addressing concepts in thermodynamics and heat transfer to replace traditional lab experiments.
• Created original simulations (1, 2) for the web using JavaScript (vanilla & p5), HTML, CSS, and jQuery.
• Updated pre-existing simulations to use Bootstrap templates & buttons for improved use on mobile devices.

Student Research Assistant – Department of Computer Science, Bucknell University, May 2016 – May 2017

• Researched the intersection between data privacy and human-computer interaction, with particular emphasis on “usable privacy” and procedures that relay intuition about these systems.
• Developed a technique for gauging users’ trust in mathematical protocols that protect confidentiality of large datasets (i.e. differential privacy).
• Designed, implemented, and deployed an experiment on Amazon’s Mechanical Turk (AMT) using a custom platform with web integration built with jQuery, JavaScript, HTML, and CSS.
• Parsed data gathered from participants and created statistical data visualizations using Python libraries such as Seaborn.

Computer Science Teaching Assistant – Department of Computer Science, Bucknell University, January 2016 – May 2016

• Led help sessions and tutored students individually or in small groups to reinforce computer science principles.
• Discussed curriculum with a computer science faculty mentor in order to coordinate instructional efforts.
• Graded weekly homework assignments and offered feedback for roughly 130 students.

PROJECT SAMPLES

Audio Processor

• Researched the Java Sound API to perform various operations on audio files, including generating digital  tones and exporting them as .wav files. Created a GUI to interface with the program and rendered the waveform visualization and Discrete Fourier Transform (DFT) of any .wav file. Project completion granted experience with algorithm optimization (to generate tones and render DFT visualizations in real time), I/O streams, byte manipulation (e.g. downsampling), and graphical user interfaces.

Twitter Bot

• Bridged the Giphy and Twitter APIs to develop an autonomous Twitter account that posts a randomly selected “word of  the day” and an accompanying GIF. Each word is assembled into a query to Giphy’s “trending” section and a callback function saves the media locally before uploading it to @Giph_Bot’s Twitter account. Project completion granted experience working with Heroku to deploy web apps that run independently, open-source Node.js packages, asynchronous programming, and Twitter’s OAuth implementation.

Custom UNIX Shell

• Developed a set of tools in the C language for a command-line interface that merged pre-existing Bash functionality with original ideas, including support for a detailed history log and randomly generated ASCII art. Later stages of development included collaboration with other students to combine functionality into a “gshell” (group shell) package. Project completion granted experience with Git repositories as well as the Bash and C languages.

Twitter Trends

• Extracted and graphically represented aggregate national Twitter sentiment (positive, negative, or neutral) of keywords (e.g. “Barack Obama,” “student loans”) over a week-long period – 1 billion Tweets. Project completion gave insight into data mining with JSON objects and developing a GUI to present information to the user using the Python language.

EDUCATION

Bucknell University – B.S.E. in Computer Science & Engineering || 2014 – May 2018

ACTIVITIES & SOCIETIES

• Association for Computing Machinery (ACM)
• Bucknell Women in Science and Engineering (B-WISE)
• Society of Women Engineers (SWE)
• Collaborative Research Experiences for Undergraduates (CREU)
• Computing Research Association (CRA)

AWARDS

• CHI 2017 Papers and Notes: Acceptance – May 2017
  • Given to the top ~20% of submissions to the ACM SIGCHI annual international conference, Papers and Notes are archival publications of original research in the field of Human Computer Interaction (HCI).
• Scholarships for Women Studying Information Security (SWSIS) – April 2017
  • The SWSIS program provides scholarships of up to $10,000 for women studying for their Bachelors and Masters degrees in fields relating to information security.
• Most Outstanding Abstract in the Area of Natural Sciences & Engineering (Susquehanna Valley Undergraduate Research Symposium) – August 2016
  • Given to the authors of the top abstract in the category of natural sciences & engineering, along with an invitation to give a 10 minute oral presentation at the symposium.
• Bucknell Women in Science and Engineering (B-WISE) Scholar – August 2014 – Present
  • Given to high-achieving women undergraduate students majoring in science or engineering, the B-WISE scholarship is renewable for $20,000 per academic year.

SKILLS

• Programming and markup languages: Python, Java, Bash, C, C++, HTML, CSS, JavaScript (+ p5, React, and Node frameworks), jQuery, Dart, Haskell, Prolog
• Typesetting and other applications: LaTeX, Netbeans, Sublime, LibreOffice, Adobe CS6 (Illustrator, Flash, Fireworks, and Dreamweaver)
• Miscellanea: Distributed version control systems (Git), Agile development methodologies (e.g. SCRUM), Unit testing (including doctest and JUnit)