I’ve been playing around with mapping steam games + their reviews based on Qwen embeddings. I revisit this project every year around this time to see what improvements I can make to make a “better” map than last year (According to my own preferences). [0/1]
One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.
This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.
The methodology feels either heavily undercooked to the point of being misleading, or heavily under explained. Why are we in 3D? At which point did dimensionality reduction hit? And if we're doing dimensionality reduction, does the "plane" really mean anything?
It's under explained. It's in 3D because of PCA. It mentions PCA all over the place, but never even says what it is, less alone briefly describe what it does.
There are a lot of more fancy ways to reduce dimensions and create visualizations like these that tend to preserve more of the higher dimensional structure.
I'm very familiar with PCA. There are so, so many ways you can apply it to this problem.
Assuming the simplest interpretation of what the analysis did, then it's saying "the variance of the biggest variance direction stays constant". Which in itself is an interesting thread to tug on, but the charts aren't how you'd present such a result.
I wish there had been a third broad language style such as arabic although perhaps Qwen doesnt use arabic.
It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.
It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.
I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.
One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.
This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.
[0]: 2025’s map: https://storage.danieltperry.me/share/steam-map/
[1]: 2026’s map (still a work in progress, both the ui and the review processing are incomplete): https://storage.danieltperry.me/share/steam-map-refresh/
https://en.wikipedia.org/wiki/Principal_component_analysis
There are a lot of more fancy ways to reduce dimensions and create visualizations like these that tend to preserve more of the higher dimensional structure.
Here's a semi recent paper discussing and comparing some of them, I didn't dig too deep to try to find a more recent or better one : https://pmc.ncbi.nlm.nih.gov/articles/PMC12453773/
Assuming the simplest interpretation of what the analysis did, then it's saying "the variance of the biggest variance direction stays constant". Which in itself is an interesting thread to tug on, but the charts aren't how you'd present such a result.
It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.
It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.