diff --git a/_posts/2024-09-30-aardvark.md b/_posts/2024-09-30-aardvark.md new file mode 100644 index 0000000..baec666 --- /dev/null +++ b/_posts/2024-09-30-aardvark.md @@ -0,0 +1,156 @@ +--- +layout: post +title: Lessons Learned from Visualizing Multimodal Data... with Aardvarks... +date: 2024-09-30 14:00:00 +categories: blog +type: blog +# Authors in the "database" can be used with just their person "key" +authors: + - lange +publication_key: 2024_vis_aardvark +# redirect_from: +# - "/event/2020/07/20/state-dashboards/" +# Use the abstract to provide a high-level overview of the blog post and main takeaways. +abstract: How do you analyze data with multiple modalities – say, images, trees and time-series? If you ask a visualization researcher, we will tell you visualizations are the solution, specifically composite visualizations! This blog shares some lessons we learned designing and developing composite visualizations to help understand cancer cell development. Regarding the aardvarks... this is clickbait, mostly. But we did get a best paper award for this project at IEEE VIS. +# Create a lead image that is <500k so that it shows up on twitter link preview +lead-image: /assets/images/posts/2024_aardvark_fight.jpeg +lead-image-alt-text: A cartoon anthropomorphized aardvark with boxing gear faces off against a gross cancer cell. The aardvark is determined to defeat the cancer cell. The cancer cell is afraid because it knows it is about to be whomped by the aardvark. +--- + +# Understanding cancer is the key to fighting it. + +Cancer is a terrible disease caused by your cells growing out of control. If we can understand exactly how cancer cells grow, move, and divide, we can develop strategies to prevent, diagnose, and treat cancer. + +A fundamental way to further our understanding is to collect data to represent how cancer cells are growing. It turns out that this is pretty hard because... + +# Cancer is complex — we need complex ✨ multimodal ✨ data to represent it. + +What is ✨ multimodal ✨ data? For us, it is data in different formats that represent different aspects of the same phenomenon. Specifically, we work with 🌌 **images**, 🌳 **trees**, and 📈 **time-series** data. + +An artificial image of an aardvark wearing a lab coat, doing science. It’s a bit in the weird AI uncanny valley style, the hands are almost correct. + +## How is this data collected? + +The datasets we worked with are from live-cell microscopy imaging; in other words, 🌌 **images** of cancer cells are recorded over time as those cells grow and divide. + + +![Animation showing a sequence of cells growing, moving, and dividing over time.]({{site.base_url}}/assets/images/posts/2024_aardvark_imaging.gif){:#aardvark_imaging} + + + + +Algorithms can track individual cells over time based on their position and other characteristics, such as size. + +![Animation showing how a single cell is tracked across the sequence of images and grows larger.]({{site.base_url}}/assets/images/posts/2024_aardvark_tracking.gif){:#aardvark_tracking} + + +Then, derived attributes based on these images can be computed, such as the cell size or the amount of a specific protein in a cell. These attributes are calculated over time, resulting in 📈 **time-series** data. + +![Animation showing a single attribute increasing over time for a single tracked cell.]({{site.base_url}}/assets/images/posts/2024_aardvark_time-series.gif){:#aardvark_time-series} + + +During these experiments cells might divide into two daughter cells. If we record these divisions, we can construct a 🌳 **tree** of cell relationships or cell lineage. + +![animation that shows one cell dividing into two, and indicating that those two divide into four in a tree of divisions.]({{site.base_url}}/assets/images/posts/2024_aardvark_tree.gif){:#aardvark_tree} + + +Now that we have the data, we need to try to understand it. It turns out that... + +# Understanding multimodal data requires us to think about all the modalities together... and this is hard. + +Each of the data modalities (🌳🌌📈) is necessary because they capture a different aspect of how cancer cells develop: + +* The 🌌 **images** show the spatial relationships between cells. +* The 📈 **time-series** data shows how cells grow and change over time. +* The 🌳 **tree** captures how cell attributes propagate across generations. + +But one thing that quickly became clear was that to fully understand the phenomenon of interest (the spread of cancer cells), we needed to synthesize all three of these modalities together. + +Right now, researchers are synthesizing or combining data modalities manually. In other words, they look at an image, then at time-series data, then back at an image, then at a tree, and mentally link data elements together. In the best case, this is tedious and taxing. In the worst case, it is impossible to relate elements from one modality to another. + +A cartoon headshot of a confused aardvark, looking straight ahead with a cloud of question mark uncertainty around it. + +So now what? Well, now is the part where the *visualization nerds* get to cheer and applaud as the hero of this story 🦸‍♀️ **visualizations** 🦸‍♂️ get to swoop in and save the day\! This is because... + +# 🍢 Composite visualizations 🍢 can show different data modalities together\! + +An artificial image of an aardvark looking at a fancy-looking visualization dashboard. The dashboard is very much a sci-fi visualization dashboard. There are some reasonable-looking bar charts, but there are also more abstract lines and shapes that probably aren’t useful in reality but give the vibe of fancy tech. Anyway, the aardvark is facing the dashboard but is actually looking back at the camera. It has an expression like, are you seeing this cool science visualization stuff? Yeah, it’s cool, and I’m cool because I’m standing next to it! + +What are 🍢 composite visualizations🍢? In short, composite visualizations combine multiple visualizations together into the same view. *(If you want a longer/better answer, Javed and Elmqvist do a great job “[Exploring the design space of composite visualization](https://www.doi.org/10.1109/PacificVis.2012.6183556).”)* + +This is great for our purposes because we have three different data modalities that are each represented best by different visualizations, and we are trying to link the elements across these modalities. + +What’s the catch? Even though they are powerful... + +# Designing composite visualizations is... (you guessed it) ...hard. Here’s our approach in **three simple steps**. + +1. **Selected a primary data modality**. Even though all data modalities are needed, specific tasks prioritize certain modalities. +2. **Choose the best visual encoding** for it. This visualization serves as the host representation. +3. **Embed secondary data modalities** as client visualizations. + +Of course, it’s a bit more complicated than this, but we’ll get to that. First, let’s give an example. + +# Here’s one of the composite visualizations we designed (this one is my 🤗 favorite 🤗). + +1. First, we select the **tree** as the **primary data type** and, by extension, name this composite visualization the tree-first visualization. +2. We **encode** this **tree** as a **node-link diagram**. +3. Then, we **embed** **the time-series** data by nesting it within the nodes of the tree and **superimpose the images** of cells above the nodes either automatically or on demand. + +![An animation that shows a schematic of the tree first diagram, first with the primary data type shown and then with the secondary data types added.]({{site.base_url}}/assets/images/posts/2024_aardvark_tree-first.gif){:#aardvark_tree-first} + + +# Here are the other two composite visualizations\! + +We call these the time-series-first visualization and the image-first visualization. + +Two composite visualization schematics. + +# What we learned. + +Ok, I have a confession. These **three simple steps** for constructing composite visualizations are hiding a lot of complexity. Specifically, step three is carrying a lot of weight here. In reality, this single step is a much more **iterative process.** + +The design space of composite visualizations can be intimidatingly large. You have to choose a visual encoding for each data type AND how those encodings get combined together. The choice of one affects the other, and there will be some back and forth while exploring this space. + +An artificial image of an aardvark dressed as a student in a classroom. The aardvark has an eager expression and has its notebook out; it is ready to learn! I don’t know how many fingers an aardvark has, but this AI image has given this aardvark three fingers on the left hand and four fingers on the right hand; well, not including thumbs, those are out of sight. + +That said, we found that the first two steps help anchor this exploration and reduce the design space. Selecting a primary data type and prioritizing a good encoding for that data type will help ensure that tasks for that primary data type can be done effectively. With that settled, figuring out how the other pieces fit into the puzzle becomes much more manageable. + +# We also made a tool! + +We didn’t just design these visualizations; we implemented them into a tool\! It’s an excellent tool\! Our collaborators like it\! I like it\! You can [try it out yourself](https://aardvark.sci.utah.edu/)\! You can watch a video that demonstrates it\! + + +
+ +
+
+ + + + +If you’re an *extreme visualization nerd,* you can even [read our research paper](https://sci.utah.edu/~vdl/papers/2024_vis_aardvark.pdf)\!\! The paper also goes into more detail about the design and theory pieces I talked about in this blog. + +I could talk about it more, but this blog is already long enough, and I’m tired of writing it, and you are probably tired of reading it. But I promise I am actually quite proud of the tool. + +# Are you still here? That must mean one of two things... + +## You are wondering what the deal is with the aardvarks. + +Well, the [research paper](https://sci.utah.edu/~vdl/papers/2024_vis_aardvark.pdf) I mentioned is titled “**Aardvark**: Composite Visualizations of Trees, Time-Series, and Images”. Aardvark is the name for the tool we made. + +*But why aardvarks?? Is it some really awesome acronym?? I bet the “v” stands for visualization\!\!* + +Sorry to burst your bubble, but it is not an acronym, I just like naming my research tools after animals (see [Ferret](https://vdl.sci.utah.edu/publications/2023_eurovis_ferret/) and [Loon](https://vdl.sci.utah.edu/publications/2021_vis_loon/)). As to why I picked aardvarks, I think they are a cool, weird animal. 🤷 + +## ...or you want to listen to us brag about our
✨🏆✨ \~ **award** \~ ✨🏆✨\! + +We are **incredibly honored** and excited to share that the [research paper](https://sci.utah.edu/~vdl/papers/2024_vis_aardvark.pdf) has received a [**Best Paper Award** at IEEE VIS 2024](https://ieeevis.org/year/2024/program/awards/awards.html)\. + +I am amped up and *just a touch* terrified to present to the whole conference this year in Florida\! I hope to see you there\! 🏖️ + +## Ok, actually three things; you want to see the “blooper” images. + +And yes, I made all of the aardvark images with AI. Specifically Adobe Firefly. These *monstrosities* are the result of asking for an image of an aardvark, loon, and ferret... + + +Four artificial images that attempt to fuse a loon, aardvark, and ferret into one animal. It is not very successful. It mostly looks like a beaver or otter swimming in water with a long neck and weird face that sort of looks like an aardvark face. \ No newline at end of file diff --git a/assets/images/posts/2024_aardvark_confused.jpg b/assets/images/posts/2024_aardvark_confused.jpg new file mode 100644 index 0000000..8a655dd Binary files /dev/null and b/assets/images/posts/2024_aardvark_confused.jpg differ diff --git a/assets/images/posts/2024_aardvark_dashboard.jpg b/assets/images/posts/2024_aardvark_dashboard.jpg new file mode 100644 index 0000000..debe0d1 Binary files /dev/null and b/assets/images/posts/2024_aardvark_dashboard.jpg differ diff --git a/assets/images/posts/2024_aardvark_fight.jpeg b/assets/images/posts/2024_aardvark_fight.jpeg new file mode 100644 index 0000000..11628f3 Binary files /dev/null and b/assets/images/posts/2024_aardvark_fight.jpeg differ diff --git a/assets/images/posts/2024_aardvark_imaging.gif b/assets/images/posts/2024_aardvark_imaging.gif new file mode 100644 index 0000000..038cdeb Binary files /dev/null and b/assets/images/posts/2024_aardvark_imaging.gif differ diff --git a/assets/images/posts/2024_aardvark_monsters.jpeg b/assets/images/posts/2024_aardvark_monsters.jpeg new file mode 100644 index 0000000..8025704 Binary files /dev/null and b/assets/images/posts/2024_aardvark_monsters.jpeg differ diff --git a/assets/images/posts/2024_aardvark_other-designs.png b/assets/images/posts/2024_aardvark_other-designs.png new file mode 100644 index 0000000..8fa111b Binary files /dev/null and b/assets/images/posts/2024_aardvark_other-designs.png differ diff --git a/assets/images/posts/2024_aardvark_science.jpg b/assets/images/posts/2024_aardvark_science.jpg new file mode 100644 index 0000000..9e79b3e Binary files /dev/null and b/assets/images/posts/2024_aardvark_science.jpg differ diff --git a/assets/images/posts/2024_aardvark_student.jpg b/assets/images/posts/2024_aardvark_student.jpg new file mode 100644 index 0000000..8612ee5 Binary files /dev/null and b/assets/images/posts/2024_aardvark_student.jpg differ diff --git a/assets/images/posts/2024_aardvark_time-series.gif b/assets/images/posts/2024_aardvark_time-series.gif new file mode 100644 index 0000000..77d29f3 Binary files /dev/null and b/assets/images/posts/2024_aardvark_time-series.gif differ diff --git a/assets/images/posts/2024_aardvark_tracking.gif b/assets/images/posts/2024_aardvark_tracking.gif new file mode 100644 index 0000000..5ff2367 Binary files /dev/null and b/assets/images/posts/2024_aardvark_tracking.gif differ diff --git a/assets/images/posts/2024_aardvark_tree-first.gif b/assets/images/posts/2024_aardvark_tree-first.gif new file mode 100644 index 0000000..3952203 Binary files /dev/null and b/assets/images/posts/2024_aardvark_tree-first.gif differ diff --git a/assets/images/posts/2024_aardvark_tree.gif b/assets/images/posts/2024_aardvark_tree.gif new file mode 100644 index 0000000..6f81254 Binary files /dev/null and b/assets/images/posts/2024_aardvark_tree.gif differ