Skip to content

Computational-Ethnography-Lab/cmap_visualization_toolkit

BranchesTags

Repository files navigation

CMAP Visualization Toolkit

An easy-to-use toolkit for visualizing patterns in qualitative data, helping researchers see and share connections between words, concepts and themes alongside in-depth accounts.

GitHub release

Table of Contents

Overview

The CMAP Visualization Toolkit offers a free suite of open-source tools to analyze and visualize text data: including fieldnotes, in-depth interview transcripts, historical documents, web-pages, and other forms of non-numeric information. It is designed for scholars working with qualitative methods, who have an interest in the possibilities for pattern analysis, data visualization, and identifying alternative explanations found in computational social science's (CSS).

The CMAP (Cultural Mapping and Pattern Analysis) tool is free, open-source and produced by the Computational Ethnography Lab at Rice University.

This toolkit implements the visualization step of the lab's end-to-end qualitative workflow — see Teaching → Workflow Steps for the general workflow and data schema.

Coming from ATLAS.ti, NVivo, or MAXQDA? The CMAP QDPX Converter turns a QDPX (REFI-QDA) export into the CMAP-schema CSV this toolkit reads — no manual reshaping.

Notebook Version & Local Installation

For an introduction, in an easy to use online version (not for sensitive data) check out the collab version here
You can find a short tutorial on using the toolkit in collab here). You can read a general description in the working paper here.

What This Toolkit Does

This notebook introduces elements of visualizing text data from qualitative sources and provides tools for:

  • Validating text data
  • Generating basic text statistics
  • Charting concepts
  • Visualizing themes
  • Drawing comparisons at the level of words, codes, concepts, and documents
  • Allowing analyses across subsets of data (e.g. examining variation by neighborhood, occupation, time)

CMAP Visualization Toolkit supports advanced analytic methods that are appropriate for computational text analysis and can be used alongside in-depth readings-- including co-occurence, clustering and embedding apporaches-- with visuals such as heatmaps, t-SNE dimensional reducation plots (like a scatter plot, with words), semantic networks, word clouds, and more. The examples are designed to work with common qualitative data sources and allow granular analysis that mirror qualitative practices (at the level of words, sentences, paragraphs), yet are scalable for large datasets produced by teams.

Sample Visualizations

Examples from this toolkit using public data on scientists' careers.

A heatmap visualizing patterned variation in interviews.
A semantic network of concepts in field notes and interviews.
A t-SNE plot for document clustering.
A word cloud with user-defined concept groups.

** Read more about the approach here**

Installation

⚠️ IMPORTANT: This toolkit requires Anaconda or Miniconda to be installed on your system. ⚠️
If you don't have Anaconda installed, please follow this video guide on how to install Anaconda.

One-Command Installation (Video Guide)

If you are using anaconda:

  • Open the anaconda program
  • Left-click 'Environments' (left side of the interface)
  • Left-click (left click the arrow next to base(root)-> cick open terminal.
  • In the terminal window that opens, copy and paste the text in steps 1-4. - For step 3 make sure to use the version for your system (windows, mac, or linux).
    • Note: some windows versions require pasting by right-clicking instead of ctrl-v

  1. Install git:

    conda install git

  2. Clone the repository:

    git clone https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit.git
cd cmap_visualization_toolkit

  3. Run the installation script:

  • This sets up the python packages needed to run the toolkit.

    For macOS/Linux:

    chmod +x install.sh
./install.sh

    For Windows (in Anaconda Prompt):

    conda create -y --name cmap_visualization_toolkit python=3.11
conda activate cmap_visualization_toolkit
pip install -r requirements.txt
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('wordnet'); nltk.download('averaged_perceptron_tagger'); print('NLTK resources downloaded successfully!')"
  1. Launch Jupyter Notebook: 
    jupyter notebook visualization_toolkit_final.ipynb

Now, you should be in the toolkit! You can run each block by clicking the run buton (triangular arrow) for each cell. The program explains what each cell does, and what can be edited for analysis/

If you get 'unable to compare versions error' check the kernel, in the top right. It should say something like visualization toolkit." ** You can also open the .ipynb file in vscode or another development environment, just check the kernel

Step-by-Step Installation

If the one-command method doesn't work, or you want more granular control at the command line, try these step-by-step commands:

First, open your terminal. Then paste the following into the command line interface in order.

# 1. Create and activate conda environment (assuming a conda version)
conda create -y --name cmap_visualization_toolkit python=3.11
conda activate cmap_visualization_toolkit
conda install git

# 2. Clone the repository
git clone https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit.git
cd cmap_visualization_toolkit

# 3. Install Jupyter (to ensure we have it before other packages)
conda install -y jupyter

# 4. Install packages from requirements.txt
pip install -r requirements.txt

# 5. Download NLTK resources
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('wordnet'); nltk.download('averaged_perceptron_tagger')"

# 6. Launch Jupyter Notebook
jupyter notebook visualization_toolkit_final.ipynb

# 6b. If you would rather use VS code or another integrated development enviornment, simply open the .ipynb file in that application.

This installation method ensures all packages are installed with the correct versions specified in the requirements.txt file.

Using Anaconda Navigator GUI

For users who prefer a visual interface:

  1. Open Anaconda Navigator:

    • Windows: Click Start menu → Anaconda Navigator
    • Mac: Open Applications folder → Anaconda Navigator
    • Linux: Open terminal and type anaconda-navigator

  2. Create a New Environment:

    • Click on "Environments" tab on the left side
    • Click "Create" button at the bottom
    • Type cmap_visualization_toolkit as the name
    • Select Python 3.11 from the dropdown
    • Click "Create" button

  3. Install Jupyter:

    • With your new environment selected, go to the "Home" tab
    • Select your new environment from the dropdown menu
    • Install Jupyter Notebook by clicking "Install"

  4. Open Terminal in Your Environment:

    • Go back to "Environments" tab
    • Click on your cmap_visualization_toolkit environment
    • Click the play button (▶) and select "Open Terminal"
    • In the terminal, run:
    # Get the code
git clone https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit.git
cd cmap_visualization_toolkit

# Install packages
pip install -r requirements.txt

# Download NLTK resources (standard language processing datasets)
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('wordnet'); nltk.download('averaged_perceptron_tagger'); print('NLTK resources downloaded successfully!')"

Using the Toolkit

First Time Setup

  1. Start Your Environment (if not already activated):

    conda activate cmap_visualization_toolkit

  2. Open the Notebook:

    jupyter notebook visualization_toolkit_final.ipynb

    If using VS Code:

    1. Open VS Code
    2. Click "File" → "Open Folder" and select the cmap_visualization_toolkit folder
    3. Find and double-click on visualization_toolkit_final.ipynb
    4. When prompted, select the cmap_visualization_toolkit kernel

  3. Run the Code:

    1. Click on the first gray box (called a "cell")
    2. Click the "Run" button (triangle symbol ▶) or press Shift+Enter
    3. Wait for it to finish (when the * symbol disappears)
    4. Move to the next cell and repeat

Regular Usage

  1. Start Your Environment:

    • IF using Navigator: Open Anaconda Navigator, click your environment, then click "▶" and "Open Terminal"
    • IF using command line: Open terminal and type conda activate cmap_visualization_toolkit, and switch to the directory with the toolkid 'cd cmap_visualization_toolkit'

  2. Open the Notebook:

    jupyter notebook visualization_toolkit_final.ipynb

  3. Run Each Section:

    1. Click on a cell
    2. Press the Run button (▶) or Shift+Enter
    3. Continue through all cells in order

  4. Save Results:

    • To save an image: Right-click on it and select "Save Image As..."
    • To copy text: Highlight it and press Ctrl+C (Windows) or Cmd+C (Mac)

Using Your Own Data

To analyze your own text data:

  1. Prepare Your Data:

    • Create a CSV file with at least a column called text
    • Optionally add a column called project to group texts
    • Save it in the data folder

  2. Change the File Path:

    • In the notebook, find the cell that loads data
    • Change the file name to your CSV file name
    • Run the cells in order

  3. Adjust Settings:

    • Word clouds: Change keywords to find specific topics
    • Networks: Adjust threshold values to show more/fewer connections
    • Heatmaps: Change clustering method (1=RoBERTa, 2=Jaccard, 3=PMI, 4=TF-IDF)

Data Structure

This is the runtime contract — the columns the notebook's loader (function/vis_tool_core.py) actually reads from your CSV. For the full conceptual schema (all fields, types, and rationale), see Teaching → Data Schema Example (CMAP).

Column Required? Type Notes
text Required string Paragraph-level content to analyze. If no text column exists, the loader falls back to the first column whose name contains text, content, or body; if none is found it stops.
codes Optional list of strings Qualitative codes. Enables code-based analyses and code filtering.
data_group Optional list of strings Document-set labels for subsetting (e.g. by data type or characteristic).
project Optional string Project label, used to subset the data to one or more projects.

Other columns from the full schema (e.g. document, position fields, word_count, doc_id) are preserved if present but are not required by the loader.

Important Notes:

  • Lists (like codes and data_group) must be proper Python lists, not strings that look like lists. The loader will attempt to convert list-looking strings, but real lists are safest.
  • If you're exporting from qualitative data analysis software, ensure you convert any code fields to proper lists.
  • The toolkit will validate your data structure and provide error messages for common issues.

Design Notes

Rationale for two setup steps in the notebook (visualization_toolkit_final.ipynb). The notebook keeps a short key-decision note at each of these steps and points here for the full explanation.

Environment Configuration

In this section, we define the core directory structure used throughout the replication project. These paths help organize:

  • raw data,
  • trained models,
  • clustering results, and
  • final outputs.

This setup also makes it easier for users to customize where intermediate results and final outputs will be saved. For example, by changing these directory names, users can create their own versions of model runs or clustering outputs without overwriting previous results.

All directories will be automatically created if they don't already exist.

Stopword Expansion and Semantic Word Family Definitions

This section defines a comprehensive list of stopwords, extending NLTK’s default stopword set with:

  • punctuation,
  • contractions,
  • common filler words, and
  • project-specific conversational terms that are semantically uninformative in analysis.

We also build a custom WORD_FAMILIES dictionary, which groups related words into unified concepts (e.g., "death", "caregiver", "memory"). This allows the model to:

  • compress synonyms and variations into semantically meaningful units,
  • reduce noise in the embedding space,
  • and support cultural/qualitative interpretation of the results.

This section also includes validation checks to:

  • Ensure no accidental overlaps between stopwords and key analytical terms,
  • Detect redundant words across families,
  • And print summaries for user verification.

These definitions are critical for interpretability in downstream visualization and clustering.

Troubleshooting

Here are solutions to common issues you might encounter:

  1. Installation Script Issues:

    • If the installation script doesn't work, try the step-by-step commands in the Step-by-Step Installation section
    • For permission issues with install.sh, run: chmod +x install.sh before executing

  2. Package Version Conflicts:

    • If you see version compatibility errors, try installing without version specifications: pip install -r requirements.txt --no-deps
    • For Mac with Apple Silicon (M1/M2/M3), you may need: pip install torch --extra-index-url https://download.pytorch.org/whl/cpu

  3. CUDA/GPU Issues with PyTorch:

    • If you encounter CUDA errors, you might need a specific torch version: pip install torch==2.1.0 --index-url https://download.pytorch.org/whl/cu118
    • For CPU-only: pip install torch==2.1.0 --index-url https://download.pytorch.org/whl/cpu

  4. Memory Errors:

    • If you get "out of memory" errors, try processing smaller batches of data
    • Close other applications to free up system memory

  5. Import Errors:

    • Make sure your directory structure is correct with the function folder at the same level as the notebook
    • Check that all packages are installed correctly

  6. Visualization Issues:

    • If plots are not displaying correctly, try running %matplotlib inline in a notebook cell
    • For interactive plots, run pip install ipywidgets and then jupyter nbextension enable --py widgetsnbextension

  7. Data Format Issues:

    • If you see errors related to data types, ensure your CSV has the correct format per the schema
    • Common issue: Make sure codes and data_group are proper lists, not strings
    • Fix: Use df['codes'] = df['codes'].apply(lambda x: ast.literal_eval(x) if isinstance(x, str) else x) to convert string representations to lists

Uninstallation

To remove the CMAP Visualization Toolkit from your system:

  1. Remove the Environment:

    # Deactivate the environment if it's currently active
conda deactivate

# Remove the environment and all its packages
conda env remove --name cmap_visualization_toolkit

  2. Delete the Code:

    # Navigate up one directory (if you're in the project directory)
cd ..

# Remove the project directory
rm -rf cmap_visualization_toolkit

  3. Clean Conda Cache (Optional):

    # Remove unused packages and caches
conda clean --all

This will completely remove all toolkit components from your system.

Training Resources

Anaconda Setup Videos

For more detailed information, refer to the Anaconda Documentation.

References

This toolkit builds on academic work combining computational text analysis with qualitative research methods (Abramson et al. 2018, 2025). For a curated topical bibliography with DOIs, see Teaching → Bibliography. Please see the lab repo for additional resources and related research papers.

Policies

License

See LICENSE.md. BSD 3-Clause License Copyright (c) 2025 Computational Ethnography Lab (Abramson et al.)

Important: If you use this software, please cite as: "Abramson, Corey and Yuhan (Victoria) Nian. 2025. CMAP Visualization Toolkit. DOI https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit."

No warranty is provided. If you want to contribute, please email corey.abramson@rice.edu.

Key Contributors

Acknowledgments

We thank all contributors to various iterations of this code for their valuable feedback, particularly the contributors above, and UC San Francisco's Medical Cultures Lab.

Data Ethics

See data_ethics.md for details on ethical considerations, including data anonymization, consent, and use restrictions.

Disclaimer

See DISCLAIMER.md for important legal and usage disclaimers.

Disclosures

  • LLMs (primarily Claude-Sonnet) were used to check for errors and help annotate code.
  • If you find errors or are interested in collaboration, contact corey.abramson@rice.edu.
  • This free software carries no warranty or guarantee.

About

open-source visualization prototype

Topics

visualization notebook text-analysis research-tool in-development teaching-tool computational-social-science bsd-3-clause mixed-methods qualitative-methods computational-ethnography

Resources

Readme

License

View license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

1 fork
Report repository

Releases

2 tags

Packages

 
 
 

Contributors

Languages