Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .image-size-overrides
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,12 @@
# images/research/big-zebrafish-figure.png
#
# Files listed here will not block commits but will still print a warning.

# Recovered from dibs-web01 in issue #33. Both genuinely warrant the
# size; could be downscaled in a separate cleanup pass:
# - raymond_poster.jpg is a 4032x3024 conference-poster photo (~2 MB
# after jpegoptim --max=85). Detail matters for legibility.
# - website_VAEGAM_fig.png is a multi-panel research figure where
# the detail in the panels is the point.
images/raymond_poster.jpg
images/website_VAEGAM_fig.png
2 changes: 1 addition & 1 deletion _layouts/home.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

<body style="background: url('/images/plab_logo_dark.svg') no-repeat center center; background-size: 80%">
{% include navbar.html %}
<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/DUSOM_Dept_Neurobio_stack.jpg", width="25%", style="float:right;padding-top: 50px"/>
<img src="/images/DUSOM_Dept_Neurobio_stack.jpg", width="25%", style="float:right;padding-top: 50px"/>
</body>
<footer>
<div style="display: block; position: fixed; bottom: 20; left: 30px">
Expand Down
2 changes: 1 addition & 1 deletion _posts/2018-12-5-incubator-award.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ post_title: "DIBS incubator award"
author: John Pearson
category: blog
---
<img style="float: left; padding-right: 30px; width: 50%" src="https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/pipelineNewpng3.png">
<img style="float: left; padding-right: 30px; width: 50%" src="/images/pipelineNewpng3.png">
[We won an Incubator Award](https://dibs.duke.edu/news/2018-2019-research-incubator-awards-announced) from the Duke Institute for Brain Sciences for Anne's work in collaboration with [Eva Naumann](https://www.neuro.duke.edu/people/faculty/eva-naumann) on real-time analysis of zebrafish data. See more information on our [research page](https://pearsonlab.github.io/research.html#work-in-progress-real-time-analysis-of-neural-data).
<br>
<br>
Expand Down
4 changes: 3 additions & 1 deletion _posts/2019-10-29-vae-preprint.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ post_title: "New preprint on bioRxiv!"
author: Anne Draelos
category: blog
---
<img style="float: right; padding-left: 20px; width: 40%" src="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/ava_preprint.png">
<!-- Original post image (ava_preprint.png) was hosted on the now-defunct
dibs-web01.vm.duke.edu and wasn't recoverable during the issue #33
migration. Restore here if a copy of the figure surfaces. -->
Jack's work on latent space analyses for animal vocalizations is [up on the bioRxiv](https://www.biorxiv.org/content/10.1101/811661v2)! Autoencoded Vocal Analysis, the Python package used to generate, plot, and analyze latent features, is freely available [on github](https://github.com/jackgoffinet/autoencoded-vocal-analysis). For more information on this project, feel free to check out our research page.
<br>
<br>
Expand Down
2 changes: 1 addition & 1 deletion _posts/2019-7-26-huang-poster.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ post_title: "Huang Fellows summer poster session"
author: Anne Draelos
category: blog
---
<img style="float: left; padding-right: 30px; width: 40%" src="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/raymond_poster.jpg">
<img style="float: left; padding-right: 30px; width: 40%" src="/images/raymond_poster.jpg">
Congratulations to Raymond Chen for his excellent work this summer, as well as his successful presentation during the Huang Fellows summer poster session. We see great things in his future research endeavors!
<br>
<br>
Expand Down
Binary file added images/DUSOM_Dept_Neurobio_stack.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Trevor.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/colorFish.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/john.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/location/admin_ent.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/location/bryan_ent.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/location/bryan_ext.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/location/ctn_ent.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/pipelineNewpng3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/raymond_poster.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/shiyang.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/vae_finch.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/website_VAEGAM_fig.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions location.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@ nav: Find Us # what shows up in the navbar at the top (do not define if you don'

We're located in the [Bryan Research Building](https://maps.duke.edu/?focus=68). This is a view of it from across Research Drive:

<img src="http://dibs-web01.vm.duke.edu/pearson/assets/images/website/bryan_ext.jpg" width="500"/>
<img src="/images/location/bryan_ext.jpg" width="500"/>

If you walk toward the building from either side, you'll find the entrance inside the breezeway in the middle:

<img src="http://dibs-web01.vm.duke.edu/pearson/assets/images/website/bryan_ent.jpg" width="500"/>
<img src="/images/location/bryan_ent.jpg" width="500"/>

Inside the lobby, turn right and enter the department offices:

<img src="http://dibs-web01.vm.duke.edu/pearson/assets/images/website/admin_ent.jpg" width="500"/>
<img src="/images/location/admin_ent.jpg" width="500"/>

Head toward the back, where you'll find the Center for Theoretical Neurobiology. The lab and John's office are inside.

<img src="http://dibs-web01.vm.duke.edu/pearson/assets/images/website/ctn_ent.jpg" width="500"/>
<img src="/images/location/ctn_ent.jpg" width="500"/>
4 changes: 0 additions & 4 deletions lychee.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,10 @@ accept = [200, 204, 206, 401, 403, 405, 429, 999]
# HTTP-checking them only tells us about the *currently deployed*
# site, not our build. Internal page-to-page links should be relative
# anyway (lychee resolves them against the local _site/ directory).
# - dibs-web01.vm.duke.edu: known dead/flaky external image host. The
# migration into the repo is tracked as Phase 5; remove this exclude
# once that migration lands so any new dead URLs are caught.
# - maxcdn.bootstrapcdn.com: stable enough that transient 5xx responses
# shouldn't fail the build.
exclude = [
"pearsonlab\\.github\\.io",
"dibs-web01\\.vm\\.duke\\.edu",
"maxcdn\\.bootstrapcdn\\.com",
# localhost references in README.md (jekyll-readme-index copies
# README into _site/ so lychee scans it).
Expand Down
6 changes: 3 additions & 3 deletions people.html
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ <h2>Principal Investigator:</h2>
<br />
<div class="row full-width">
{% include person.html
image="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/john.png"
image="/images/john.png"
site="http://jmxpearson.com" github="https://github.com/jmxpearson"
email="john.pearson@duke.edu"
linkedin="https://www.linkedin.com/in/jmxpearson" name="John Pearson"
Expand All @@ -66,7 +66,7 @@ <h2>Graduate Students:</h2>
<br />
<div class="row full-width">
{%
include person.html image="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/Trevor.jpg"
include person.html image="/images/Trevor.jpg"
github="https://github.com/Trevorules" email="trevor.alston@duke.edu"
linkedin="www.linkedin.com/in/trevor-alston" name="Trevor Alston" desc="Trevor is a sixth-year Neurobiology graduate student at Duke University.
He holds two Biomedical Engineering Degrees; he received his BS degree from Rutgers University and his MS degree from Northwestern University.
Expand Down Expand Up @@ -95,7 +95,7 @@ <h2>Graduate Students:</h2>
name="David St-Amand" title="Neurobiology Graduate Student" desc="David is a third-year graduate student in Neurobiology. He graduated from McGill University, where he received his BS in Psychology and his MS in Neuroscience. His research investigates how efficient coding models can help us understand how information is computed in the retina. Outside the lab, he likes to work out and play games.
<br /><br /><br /><br />" %}
{%
include person.html image="https://dibs-web01.vm.duke.edu/pearson/assets/images/website/shiyang.jpg"
include person.html image="/images/shiyang.jpg"
github="https://github.com/WaAaaAterfall" email="shiyang.pan@duke.edu"
linkedin="https://www.linkedin.com/in/shiyangpan"
site="https://waaaaaterfall.github.io"
Expand Down
8 changes: 4 additions & 4 deletions research.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ nav: Research
Together with [Eva Naumann's](https://www.neuro.duke.edu/research/faculty-labs/naumann-lab) lab, we've developed *[improv](https://github.com/project-improv/improv)* ([paper](https://www.biorxiv.org/content/10.1101/2021.02.22.432006v1)), a software platform for designing and orchestrating adaptive experiments. By analyzing data in real time, we can measure, model, and manipulate neural activity in response to new data. We've shown how these tools, in conjunction with holographic photostimulation, could in principle map functional connectivity of large circuits in a few hours ([paper](https://proceedings.nips.cc/paper/2020/file/531d29a813ef9471aad0a5558d449a73-Paper.pdf), [expanded version](https://arxiv.org/abs/2007.13911)). More recently, we've worked on methods for fast dimensionality reduction and modeling of neural populations in real time ([paper](https://arxiv.org/abs/2108.13941)).
<div class="row">
<figure>
<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/colorFish.png" class="img-responsive">
<img src="/images/colorFish.png" class="img-responsive">
<figcaption>
A whole zebrafish brain activity map, showing the distribution of motion-sensitive neurons, color-coded to show the preferred motion direction.
</figcaption>
</figure>
</div>
<div class="row">
<figure>
<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/pipelineNewpng3.png" class="img-responsive">
<img src="/images/pipelineNewpng3.png" class="img-responsive">
<figcaption>
Concept for the closed-loop pipeline. Neural data from the zebrafish are collected in the form of images, preprocessed, and analyzed in real-time. Targets for optical stimulation are then chosen based on the results of this analysis, creating adaptive experiments that test causal hypotheses.
</figcaption>
Expand All @@ -31,7 +31,7 @@ Vocalization is a complex behavior that underlies vocal communication and vocal

<div class="row">
<figure>
<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/vocal/vae_finch.png" class="img-responsive">
<img src="/images/vae_finch.png" class="img-responsive">
<figcaption>
a. Syllable VAE: Segmented syllables from adult Zebra Finch song are projected to a low-dimensional space, then reconstructed from that space using a VAE. b. Shotgun VAE: The VAE is trained on 20ms segments of adult Zebra Finch song to model variability on a millisecond duration. Visualized are songs projected into the latent space using these shorter segments.
</figcaption>
Expand All @@ -53,7 +53,7 @@ In recent work, we've explored the idea of using variational autoencoder (VAE) m

<div class="row">
<figure>
<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/fmri/website_VAEGAM_fig.png" class="img-responsive">
<img src="/images/website_VAEGAM_fig.png" class="img-responsive">
<figcaption>
A) VAE-GAM Model Schematic: brain volumes with signal of interest are compressed to a lower dimensional representation using encoder network. Sampled latents are then fed through decoder network to yield a base map and separate spatial effect maps. Each effect map is scaled by a potentially non-linear gain modelled using a Gaussian Process. Variance is modeled separately on a per voxel basis. B) Sample Effect Maps for VAE-GAM and GLM: effect maps for a visual stimulation task dataset analysed using the proposed VAE-GAM approach (top) vs. the tranditional (GLM) approach.
</figcaption>
Expand Down
149 changes: 149 additions & 0 deletions scripts/recover_dibs_web01.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
#!/usr/bin/env python3
"""Download images currently hosted on dibs-web01.vm.duke.edu into the repo.

Tracked by issue #33. Run from the repo root:

python3 scripts/recover_dibs_web01.py

For each URL, tries the live host first, then falls back to the most
recent Wayback Machine snapshot. Files land under images/ with the
target path shown below. After this finishes, commit the new images
and a separate URL-replacement commit will swap the source files to
the local paths.

Exit code is the number of URLs that couldn't be recovered.
"""

from __future__ import annotations

import os
import sys
import time
import urllib.request
import urllib.error
import urllib.parse


# Mapping of dibs-web01 URLs to the local path we want them at.
# Picked to mirror the categorization in the URL itself.
URLS: dict[str, str] = {
# Building entrances (location.md)
"http://dibs-web01.vm.duke.edu/pearson/assets/images/website/admin_ent.jpg":
"images/location/admin_ent.jpg",
"http://dibs-web01.vm.duke.edu/pearson/assets/images/website/bryan_ent.jpg":
"images/location/bryan_ent.jpg",
"http://dibs-web01.vm.duke.edu/pearson/assets/images/website/bryan_ext.jpg":
"images/location/bryan_ext.jpg",
"http://dibs-web01.vm.duke.edu/pearson/assets/images/website/ctn_ent.jpg":
"images/location/ctn_ent.jpg",
# People photos (people.html)
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/john.png":
"images/john.png",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/Trevor.jpg":
"images/Trevor.jpg",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/shiyang.jpg":
"images/shiyang.jpg",
# Duke School of Medicine Neurobio Dept logo (_layouts/home.html)
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/DUSOM_Dept_Neurobio_stack.jpg":
"images/DUSOM_Dept_Neurobio_stack.jpg",
# Research figures (research.md)
"https://dibs-web01.vm.duke.edu/pearson/assets/images/fmri/website_VAEGAM_fig.png":
"images/research/website_VAEGAM_fig.png",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/vocal/vae_finch.png":
"images/research/vae_finch.png",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/colorFish.png":
"images/research/colorFish.png",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/pipelineNewpng3.png":
"images/research/pipelineNewpng3.png",
# Blog post images (_posts/*)
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/ava_preprint.png":
"images/blog/ava_preprint.png",
"https://dibs-web01.vm.duke.edu/pearson/assets/images/website/raymond_poster.jpg":
"images/blog/raymond_poster.jpg",
}

WAYBACK_API = "https://archive.org/wayback/available?url="
USER_AGENT = "Mozilla/5.0 (compatible; pearsonlab-recover-script)"
TIMEOUT = 30


def fetch(url: str) -> bytes | None:
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
if resp.status == 200:
data = resp.read()
if len(data) > 100: # filter out tiny error pages
return data
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError):
pass
return None


def wayback_lookup(url: str) -> str | None:
"""Find the most recent Wayback snapshot URL for a given URL."""
import json
api_url = WAYBACK_API + urllib.parse.quote(url, safe="")
try:
req = urllib.request.Request(api_url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
payload = json.loads(resp.read())
snapshot = payload.get("archived_snapshots", {}).get("closest")
if snapshot and snapshot.get("available"):
return snapshot["url"]
except Exception:
pass
return None


def recover(url: str, dest: str) -> bool:
print(f"\n[*] {url}")
print(f" -> {dest}")
os.makedirs(os.path.dirname(dest), exist_ok=True)

# Try live host first.
data = fetch(url)
source = "dibs-web01" if data else None

# Fall back to Wayback.
if not data:
snapshot_url = wayback_lookup(url)
if snapshot_url:
print(f" trying Wayback: {snapshot_url}")
data = fetch(snapshot_url)
if data:
source = "wayback"

if not data:
print(f" FAILED: not reachable on live host or Wayback")
return False

with open(dest, "wb") as f:
f.write(data)
size_kb = len(data) / 1024
print(f" saved ({size_kb:.0f} KB, source: {source})")
return True


def main() -> int:
failed: list[str] = []
for url, dest in URLS.items():
if not recover(url, dest):
failed.append(url)
time.sleep(0.5) # be polite

print(f"\n{'='*60}")
print(f"Done. Recovered {len(URLS) - len(failed)}/{len(URLS)} images.")
if failed:
print(f"\nFailed URLs ({len(failed)}):")
for url in failed:
print(f" - {url}")
print("\nFor these, options are:")
print(" - Ask the original author for the file")
print(" - Manually paste a screenshot or alternate version into the repo")
print(" - Remove the reference from the source HTML/markdown")
return len(failed)


if __name__ == "__main__":
sys.exit(main())
Loading