x1 | y1 | x2 | y2 | x3 | y3 | x4 | y4 |
---|---|---|---|---|---|---|---|
10 | 8.04 | 10 | 9.14 | 10 | 7.46 | 8 | 6.58 |
8 | 6.95 | 8 | 8.14 | 8 | 6.77 | 8 | 5.76 |
13 | 7.58 | 13 | 8.74 | 13 | 12.74 | 8 | 7.71 |
9 | 8.81 | 9 | 8.77 | 9 | 7.11 | 8 | 8.84 |
11 | 8.33 | 11 | 9.26 | 11 | 7.81 | 8 | 8.47 |
14 | 9.96 | 14 | 8.10 | 14 | 8.84 | 8 | 7.04 |
6 | 7.24 | 6 | 6.13 | 6 | 6.08 | 8 | 5.25 |
4 | 4.26 | 4 | 3.10 | 4 | 5.39 | 19 | 12.50 |
12 | 10.84 | 12 | 9.13 | 12 | 8.15 | 8 | 5.56 |
7 | 4.82 | 7 | 7.26 | 7 | 6.42 | 8 | 7.91 |
5 | 5.68 | 5 | 4.74 | 5 | 5.73 | 8 | 6.89 |
Introduction
- Learning outcomes:
- Understand idea/aims/advantages/steps of interactive data analysis
- Understand basic structure of Shiny apps (components)
- Quick overview of our ESS app
- Discuss MVP and workflow
Sources: Bertini (2017), Wickham (2021) and Fay et al. (2021)
1 Interactive data analysis
- Source(s): Bertini (2017) and others.
1.1 Why?
- “From Data Visualization to Interactive Data Analysis” (Bertini 2017)
- Main uses of data visualization: Inspirational, explanatory and analytical1
- “data analysis […] can help people improve their understanding of complex phenomena”
- “if I understand a problem better, there are higher chances I can find a better solution for it”
1.2 Interactivity data visualization history
- “interactive data visualization enables direct actions on a plot to change elements and link between multiple plots” (Swayne 1999) (Wikipedia)
- Interactivity revolutionizes the way we work with and how we perceive data (cf. Cleveland and McGill 1984)
- Started ~last quarter of the 20th century, PRIM-9 (1974) (Friendly 2006, 23, see also Cleveland and McGill, 1988, Young et al. 2006)
- We have come a long way… John Tukey on prim9
- Interactivity allows for…
- …making sense of big data (more dimensions)
- …exploring data
- …making data accessible to those without background
- …generating interactive “publications”
1.3 How Does Interactive Data Analysis Work?
- Figure 1 outlines process underlying interactive data analysis
- Loop
- start with loosely specified goal/problem (Decrease crime!)
- translate goal into one or more questions (What causes crime?)
- gather, organize and analyze the data to answer these questions (Gather data on crime and other factors, model and visualize it)
- generate knowledge and new questions and start over
- Loop
![](resources/02-bertinin-2.webp)
1.4 Steps of interactive data analysis
- Defining the problem: What problem/goal are you trying to solve/reach through interactive data analysis?
- Generating questions: Translate high-level problem into number of data analysis questions
- Gathering, transforming and familiarizing with the data, e.g., often slicing, dicing and aggregating the data and to prepare it for the analysis one is planning to perform.
- Creating models out of data (not always): using statistical modeling and machine learning methods to summarize and analyze data
- Visualizing data and models: results obtained from data transformation and querying (or from some model) are turned into something our eyes can digest and hopefully understand.
- Simple representations like tables and lists rather than fancy charts are perfectly reasonable visualization for many problems.
- Interpreting the results: once results have been generated and represented in visual format, they need to be interpreted by someone (crucial step!)
- complex activity including understanding how to read the graph, understanding what graph communicates about phenomenon of interest, linking results to questions and pre-existing knowledge of problem (think of your audience!)
- Interpretation heavily influenced by pre-existing knowledge (about domain problem, data transformation process, modeling, visual representation)
- Generating inferences and more questions: steps above lead to creating new knowledge, additional questions or hypotheses
- Outcome: not only answers but also (hopefully better, more refined) questions
1.5 Important aspects of data analysis & quo vadis interaction?
- Process not sequential but highly iterative (jumping back/forth between steps)
- Some activities exclusively human, e.g., defining problems, generating questions, etc.
- Visualization only small portion of process and effectiveness depends on other steps
- Interaction: all over the place… every time you tell your computer what to do (and it returns information)
- Gather and transform the data
- Specify a model and/or a query from the data
- Specify how to represent the results (and the model)
- Browse the results
- Synthesize and communicate the facts gathered
- Direct manipulation vs. command-Line interaction: WIMP interfaces (direct manipulation, clicks, mouse overs, etc.,) are interactive but so is command line
- You can let users type!
- Audience: what (interaction) skills and pre-knowledge do the have? (domain knowledge, statistics, graphs)
1.6 Challenges of Interactive Visual Data Analysis
- Broadly three parts… (Bertini 2017)
- Specification (Mind → Data/Model): necessary to translate our questions and ideas into specifications the computer can read
- Shiny allows non-coders to perform data analysis, but requires R knowledge to built apps
- But even simpler tools out there
- Representation (Data/Model → Eyes)
- next step is to find a (visual) representation so users can inspect and understand them
- “deciding what to visualize is often equally, if not more, important, than deciding how to visualize it”
- “how fancy does a visualization need to be in order to be useful for data analysis?”
- “most visualization problems can be solved with a handful of graphs”
- really hard to use, tweak, and combine graphs in clever/effective/innovative ways
- Interpretation (Eyes → Mind)
- “what does one need to know in order to reason effectively about the results of modeling and visualization?”
- “Are people able to interpret and trust [your shiny app]?”
2 Why visualize?
2.1 Anscombes’s quartet (1)
- Table 1 shows results from a linear regression based on Anscombe’s quartet (Anscombe 1973) often used to illustrate the usefulness of visualization
- Q: What do we find here?
y1 (Set 1) | y2 (Set 2) | y3 (Set 3) | y4 (Set 4) | |
---|---|---|---|---|
(Intercept) | 3.000 | 3.001 | 3.002 | 3.002 |
(1.125) | (1.125) | (1.124) | (1.124) | |
x1 | 0.500 | |||
(0.118) | ||||
x2 | 0.500 | |||
(0.118) | ||||
x3 | 0.500 | |||
(0.118) | ||||
x4 | 0.500 | |||
(0.118) | ||||
Notes: some notes... |
2.2 Anscombes’s quartet (2)
- Table 2 displays Anscombe’s quartet (Anscombe 1973), a dataset (or 4 little datasets)
- Q: What does the table reveal about the data? Is it easy to read?
2.3 Anscombes’s quartet (3)
- Figure 2 finally visualizes the data underlying those data
- Q: What do we see here? What is the insight?
![](02-introduction_files/figure-html/fig-anscombe-1.png)
2.4 The Datasaurus Dozen
- Figure 3 displays the datasaurus dozen as animated by Tom Westlake (see here, original by Alberto Cairo)
- Q: What do we see here? What is the insight?
![](02-introduction_files/figure-html/fig-datasaurus-own-1.gif)
3 Shiny
3.1 What is Shiny?
- History of Shiny: Joe Cheng: The Past and Future of Shiny2
- Popularity: Shiny
(vs. Ggplot2
, dplyr
)
- A web application framework for R to turn analyses into interactive web applications.. what does that mean?
- The userinterface is a webpage
- On this webpage you can manipulate things
- Behind the webpage there is a computer (your computer or a server)
- That computer/server runs R and the R script of the webapp
- When you change something on the webpage, the information is send to the computer
- Computer runs the script with the new inputs (input functions)
- Computer sends back outputs to the webpage (output functions)
3.2 Components of a Shiny app
- As depicted in Figure 4, a user interacts with a server on which the shinyapp/website is hosted
- A Shiny app has two components, the user interface (UI) and the server, that are passed as arguments to the
shinyApp()
which creates a Shiny app from this ui/server pair
- A Shiny app has two components, the user interface (UI) and the server, that are passed as arguments to the
![](resources/02-compontents-shiny-app.png)
3.3 Pro & contra Shiny
3.3.1 Pros of R Shiny:
- Fast Prototyping: Shiny is excellent for quickly turning ideas into applications; easy to use even for non-seasoned programmers
- Interactivity: lets you build interactive web apps, enhancing user engagement and experience (dashboards!)
- Integration with R Ecosystem: integrates seamlessly with R’s vast open-source ecosystem (see shiny for python)
- Statistical Modeling and Visualization: allows for complex statistical modeling and visualizations within your app
- No Need for Web Development Skills: Create web apps with R code alone (no need for HTML, CSS, or JavaScript)
- Reactivity: fairly simple to create applications that automatically update in response to user inputs
- Sharing and Publishing: apps can be easily published and shared (e.g., Shinyapps.io or shinylive)
3.3.2 Cons of R Shiny:
- Performance: apps run on top of R, an interpreted language, which can cause performance issues
- Single-threaded: R (and by extension Shiny) is single-threaded, which can also cause performance issues (see here).
- Complexity: Shiny’s basics are easy but mastering intricacies of reactivity is more challenging
- Data Gathering and Saving: It can be challenging to use Shiny for gathering and saving data to the database.
- Maintenance Cost: cost of maintaining a Shiny application over time can be high (now we have shinylive!)
- Software Dependencies: Certain Shiny applications may have many software dependencies, which could potentially lead to issues down the line
5 Your (first) Shiny app
- Below you will create you first app and we’ll use the opportunity to discuss the basic components of a shiny app (see analogous example here).
- Install the relevant packages:
- Create a directory with the name of your app “myfirstapp” in your working directory.
- Create an rscript file in Rstudio and save it in the working directory with the name
app.R
. - Copy the code below and paste it into your
app.R
script (UPDATE).
- You can run and stop the app by clicking Run App (Figure 6) button in the document toolbar.
6 Minimum viable product (MVP)
- …useful concept when building apps (see Figure 7)!
![](resources/MVP.png)
- “version […] with just enough features to be usable by early customers” to collect feedback (Wikipedia)
- “Making things work before working on low-level optimization makes the whole engineering process easier” (Fay et al. 2021)
- The “UI first” approach: often the safest way to go (Fay et al. 2021)
- Agreeing on specifications: helps everybody involved in the application to agree on what the app is supposed to do, and once the UI is set, there should be no “surprise implementation”
- Organizing work: “It’s much easier to work on a piece of the app you can visually identify and integrate in a complete app scenario”
- But…
- ..we “follow” same strategy, slowly building out our shiny app, adding features & complexity
7 Workflow: Development, debugging and getting help
- See discussions of workflow in Wickham (2021, Ch. 5, 20.2.1)
- Three important Shiny workflows:
- Basic development cycle of creating apps, making changes, and experimenting with the results.
- Debugging, i.e., figure out what’s gone wrong with your code/brainstorm solutions
- Writing reprexes, self-contained chunks of code that illustrate a problem (essential for getting others’ help)
- Below development WF, debugging later on
7.1 Development workflow
- Creating the app: start every app with the same lines of R code below (
Shift + Tab
or in menueNew Project -> Shiny Web Application
) - Seeing your changes: you’ll create a few apps a day (really?!?), but you’ll run apps hundreds of times, so mastering the development workflow is particularly important
- Write some code.3
- Launch the app with
Cmd/Ctrl + Shift + Enter
. - Interactively experiment with the app.
- Close the app.
- Go back to 1.
7.1.1 Rstudio & Shiny: A few tips
- Controlling the view: Default is a pop-out window but you can also choose
Run in Viewer Pane
andRun External
. - Document outline: Use it for navigation in your app code (
Cntrl + Shift + O
) - Using/exploring other apps: Inspect that app code, then slowly delete parts you don’t need
- Rerun app to see whether it still works after each deletion
- if only interested in UI, delete everything in within server function:
server <- function(input, output, session) {delete everything here}
- Important: Search for dependencies (that can sometimes be delete), e.g., search for
www
folder- also image links with
src
orpng
,jpg
- also image links with
7.2 Debugging workflow
- Guaranteed that something will go wrong at the start
- Cause is mismatch between your mental model of Shiny, and what Shiny actually does
- We need to develop robust workflow for identifying and fixing mistakes
- Three main cases of problems: (1) Unexpected error, (2) No error but incorrect values; (3) Correct values but not updated
- Use traceback and interactive debugger
- Use interactive debugger
- Problem unique to Shiny, i.e., R skills don’t help
- See Wickham (2021, Ch. 5.2, link) for explanations and examples
References
Footnotes
Inspirational. The main goal here is to inspire people. To wow them! But not just on a superficial level, but to really engage people into deeper thinking, sense of beauty and awe. Explanatory. The main goal here is to use graphics as a way to explain some complex idea, phenomenon or process. Analytical. The main goal here is to extract information out of data with the purpose of answering questions and advancing understanding of some phenomenon of interest.↩︎
Joe Cheng is the Chief Technology Officer at RStudio and was the original creator of the Shiny web framework, and continues to work on packages at the intersection of R and the web.↩︎
Automated testing: allows you to turn interactive experiments you’re running into automated code, i.e., run tests more quickly and not forget them (because they are automated). Requires more initial investment.↩︎