Yes, I always use `du . -h 2>/dev/null | sort -k1hr | head -nN` where that last N is however many directories I want to show. I run this regularly and have for a very long time precisely because of the issue that the post talks about.
I just ran it for the first time... I've run windirstat many times including in the last two weeks and yes, this is very fast. According to its timing it took 3.63 seconds to scan my drive... I'm not sure if it really took almost 3 minutes for windirstat, but it did take a long time, so it's plausible. More importantly, ~ 4 seconds is much easier to wait for.
windirstat currently doesn't take 3min to index 500k+ files on an 8x8tb drive raidz2 that's 90% full. When I stopped using windoratat 3ish years ago it took a few minutes to index the same array.
I think you might he using an old version, or have an issue with your storage.
It is - it ‘cheats’ by reading off the NTFS MFT, similarly to another indispensable piece of Windows software Everything Search, both of which I miss keenly on my underprovisioned work Mac.
I recently ran `df` and was surprised to see that I'd used more than half of the two terabyte SSD in my laptop. So, being apparently atypically proactive, my first action was delete all of the target directories in my Rust projects and was equally surprised to see that it freed up nearly 10% of my disk usage (~200 GB)!
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
This can be an important reason to enable compression (on some file systems it “solves” the problem) and why virtual machine file format can be important.
I think about this problem often in life because of my tendency to procrastinate. Only the long term effects that cause me serious paranoia get any real attention (failing to meet social expectations, major health issues, large-figure finance problems). It’s absolutely necessary to define the constraints before they define you. This requires a lot of forward thinking and effort but considering how little time we get on this earth… worth it
It is so wild that someone would let Claude index all of the files on their computers for such a bane request. We are really turning into the people on WALL·E
I have less than one terabyte of disk space, and less than 10% of the full capacity is in use, so it is closer to being empty. (The hard drives they sell have more capacity than I require.) And, there are files that I do not use (some of which come with the operating system, such as PulseAudio).
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
And i am wondering what do people even fill their storage with, and why do they need so much. In my case it is ~200-300 out of 1TB on 2 laptops, 37 out of 128gb on phone. And i don't even actively manage it.
I have one single video game I play that takes up around 650GB, so I have a dedicated drive for it. And between 1 and 2 TB of movies, TV, and photos. Everything else is ephemeral/scratch space for whatever I'm working on.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
I have 2TB filled and I need more. I make music so I need a lot of storage for my instruments and samples. This is measured in tens and often hundreds of gigabytes. I also play games. Games' sizes range wildly from 10 to 200 GBs. I also do gamedev. The textures get big and get duplicated between projects. Programming (npm modules for example), add to this personal data, media archives and 2TB is not enough
Slack has value, though. Your full disk is likely to be very fragmented, making file access slow. A CPU running a full throttle cannot deal with spikes in demand. full RAM means less cache space available. And these issues get asymptotically worse as you approach 100% utilization (which can have catastrophic effects in organizations which are too heavily optimized for high utilization).
Delete is expensive. "Am I positive I'll never need this again?"
As a consequence, "cheap" actions (saves) accumulate, "expensive" ones (deletes) accrue until cheap becomes expensive as space becomes tight.
Ultimately, options become:
- Find criteria for rapidly assessing what data can be deleted.
- Purchase cheap archival storage and migrate old data to that.
- A media or device crash induces loss of arbitrarily-selected data.
On Linux/macOS you can use `du` from the command line (or ask an AI to make a convenience wrapper for you).
Thats a pretty bold claim, but is it really that much faster?
I think you might he using an old version, or have an issue with your storage.
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look.
I'd never have never found these myself.
Problem solved. For now!
https://github.com/Byron/dua-cli
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
Filling an ssd creates the one situation where nand wear out is a problem.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
I bought 16TB disk because I needed that much storage.
If I were using, say, only 10TB, I would be wasting 6TB.
I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM.
And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs.
Anything else would mean oversizing and overspending.