dir2lz4 anyone?

[M. Eerie]

Are you looking to further optimize your Porteus modules? Starting with kernel v. 3.18, you can leverage the LZ4 compression algorithm, which offers a fantastic combination of blazing-fast decompression (even better than zstd) and lightweight resource usage—perfect for systems with limited resources.

Benefits of Using LZ4:

• Speed: Modules compress and decompress at remarkable speeds, making boot times lightning fast.
• Eco-friendly: While maintaining high performance, LZ4 delivers reasonable compression ratios combined with efficient decompression that work excellently in embedded or minimal setups due to its low CPU usage.
• Simplicity: Reconverting all base modules took as little as 10 seconds in my Ryzen 7 machine, producing a final base dir size of 799.8 MB for the XFCE4 newer version.

HOW TO CREATE YOUR OWN dir2lz4 script:

Code: Select all

sudo cp /usr/local/bin/dir2xzm /usr/local/bin/dir2lz4
sudo sed -i '2s/dir2xzm/dir2lz4/' /usr/local/bin/dir2lz4
sed -i 's/\$COMP/-b 256k -comp lz4/' /usr/local/bin/dir2lz4

Save your changes or alternatively copy the file /usr/local/bin/dir2lz4 to your $PORTEUS/rootcopy folder:

Code: Select all

sudo mkdir -p $PORTDIR/rootcopy/usr/local/bin
sudo cp /usr/local/bin/dir2lz4 $PORTDIR/rootcopy/usr/local/bin/dir2lz4
sudo chmod +x $PORTDIR/rootcopy/usr/local/bin/dir2lz4

That's it. From now on, you can just:

Code: Select all

dir2lz4 /your/directory/tree/to/be/converted/into/a/module your-lz4-module.xzm

BONUS TRACK:
============

Script to reconvert all your modules to LZ4 algorithm:

Code: Select all

#!/bin/bash
# Converts all .xzm modules in $PORTDIR/base to LZ4 algorithm
# Author: M. Eerie @ forum.porteus.org

[[ $EUID -ne 0 ]] && { echo "This script must be run as root."; exit 1; }

read -p "Enter the path to the input directory: " INPUT_DIR

[[ ! -d "$INPUT_DIR" ]] && { echo "The specified directory does not exist. Exiting."; exit 1; }

OUTPUT_DIR="/tmp/LZ4mods/"
TEMP_DIR="/tmp/LZ4extract" # Temporary extraction folder

mkdir -p "$OUTPUT_DIR" "$TEMP_DIR"

    # Create a new SquashFS with LZ4 compression*, preserving permissions *Supported since kernel v3.18
    # https://unix.stackexchange.com/questions/256898/squashfs-minimal-compression
ls "$INPUT_DIR"/*.xzm 2>/dev/null | while read -r FILE; do
if [[ -f "$FILE" ]]; then
    BASENAME="${FILE%.xzm}"
    TEMP_EXTRACT_DIR="$TEMP_DIR/${BASENAME##*/}"
    OUTPUT_FILE="$OUTPUT_DIR/${BASENAME##*/}.xzm"

    mkdir -p "$TEMP_EXTRACT_DIR"
    unsquashfs -f -d "$TEMP_EXTRACT_DIR" "$FILE"
    mksquashfs "$TEMP_EXTRACT_DIR" "$OUTPUT_FILE" -comp lz4
    rm -rf "$TEMP_EXTRACT_DIR"
    echo "Converted: $FILE -> $OUTPUT_FILE"
  else
    echo "No '.xzm' files were found in $INPUT_DIR."
  fi
done

rm -rf "$TEMP_DIR"
echo "Conversion completed. The converted files are in $OUTPUT_DIR."

For the record, I’m writing this from my freshly reconverted version of Nemesis XFCE4, which uses LZ4 modules.

[AcnapyxoB]

It's pretty useful. The only drawback is the large size

for example 000-kernel-6.14-64bit from 156.5 became 243.0 MiB

[M. Eerie]

large size

Yes, in the end, the compression algorithm is a trade-off between size and speed.

The idea is to complete the set of options. On one hand, we have xz, which offers greater compression at the cost of slower performance, and lz4, which is incredibly fast but provides minimal compression. Somewhere in between the two, we can place zstd.

When it comes to starting up (decompressing files), lz4 is undoubtedly the fastest of them all. It makes you feel like it’s a real system.

On the other hand, its almost imperceptible compression speed makes it very suitable for saving sessions/changes—especially when you’re in a hurry.

[AcnapyxoB]

M. Eerie thanks for the info.

It makes you feel like it’s a real system

Really lz4 compression/decompression speed is impressive.

[M. Eerie]

Some figures after doing quick tests:

Code: Select all

[guest@porteus ~]$

...
{ time mksquashfs "$EXTRACT_DIR" "$OUT_FILE" -comp lz4; } 2>> test.log
...

[guest@porteus ~]$ grep "real" test.log | awk '{split($2, t, "m"); split(t[2], s, "s"); gsub(",", ".", s[1]); sum += t[1] * 60 + s[1]} END {print sum}'
[guest@porteus ~]$ 2.033

[guest@porteus ~]$

...
{ time mksquashfs "$EXTRACT_DIR" "$OUT_FILE" -b 256K -comp lz4 -Xhc; } 2>> test-Xhc.log
...

[guest@porteus ~]$ grep "real" test-Xhc.log | awk '{split($2, t, "m"); split(t[2], s, "s"); gsub(",", ".", s[1]); sum += t[1] * 60 + s[1]} END {print sum}'
[guest@porteus ~]$ 29.056

Results for Nemesis XFCE4 2025-04-01 base dir lz4 compression test:

Original Size: -comp xz -Xbcj x86 -b 256K (dir2xzm) 465,1MB
lz4 standard compression (-comp lz4 (-b 128K)): 2.033 (2 seconds) 799,8MB
lz4 high compression (-comp lz4 -Xhc -b 256K: 29.056 (29 seconds) 665,6MB

[rych]

I've had a chat with DeepSeek about this.

I always deploy my modules from already fast USB3 to often even faster internal NVMe SSD. Or recently, I simply "installed" Porteus/X onto the internal SSD, with larger speed and capacity. So for me then LZ4 would make sense.

But if you read your modules from a slow and limited USB, or from a fast but still limited RAM, or rarely used modules that only take space and used once per year -- then Zstd compression is still recommended.

When using **SquashFS** (a block-based, compressed read-only file system), the choice between LZ4 and Zstd depends on how the file system's **block-wise compression** interacts with your storage medium, access patterns, and workload. Here's a structured analysis:

---

### **1. How SquashFS Works**
- **Block Compression**:
SquashFS divides data into fixed-size blocks (default: 128–256 KB) and compresses each block individually.
- **Random Access**: Only the blocks required to fulfill a read request are loaded and decompressed.
- **Metadata Efficiency**: Directories, inodes, and file structures are compressed, minimizing seeks.

- **Key Takeaway**:
Total archive size matters less than **per-block I/O and decompression time**, since only accessed blocks are processed.

---

### **2. LZ4 vs. Zstd in a Block-Based System**
#### **Factors to Consider**
- **Compression Ratio**:
- Zstd typically achieves **~10–20% better compression** than LZ4, reducing per-block size.
- Example: If LZ4 compresses a block to 100 KB, Zstd might compress it to 80 KB.

- **Decompression Speed**:
- LZ4: **~2–5 GB/s** (faster per-block decompression).
- Zstd: **~1–2 GB/s** (slower, but varies with compression level).

- **Storage Medium Speed**:
- **HDD**: Slow read speeds (~100–200 MB/s) make smaller blocks (Zstd) advantageous.
- **SSD/NVMe**: Fast read speeds (~500 MB/s–7 GB/s) reduce I/O bottlenecks, favoring LZ4’s speed.

---

### **3. Performance Scenarios**
#### **Case 1: HDD or Slow Storage**
- **Zstd Advantage**:
- Smaller blocks mean fewer physical reads and less seek time.
- Example: Reading 80 KB (Zstd) vs. 100 KB (LZ4) per block saves ~20% I/O time.
- **Result**: Zstd likely outperforms LZ4 due to reduced I/O overhead, despite slower decompression.

#### **Case 2: SSD/NVMe or In-Memory Storage**
- **LZ4 Advantage**:
- Fast I/O minimizes the penalty of larger blocks.
- Example: On an NVMe SSD (3.5 GB/s), reading 100 KB (LZ4) takes **0.03 ms**, while decompression takes **0.01 ms**.
- **Result**: LZ4’s faster decompression dominates total latency, making it faster overall.

#### **Case 3: Mixed Workloads (Random + Sequential Access)**
- **LZ4 Shines**:
- If data is accessed repeatedly (e.g., cached blocks), LZ4’s decompression speed reduces CPU load.
- Example: Databases or real-time applications benefit from LZ4’s low-latency decompression.

---

### **4. When to Prefer LZ4 for SquashFS**
- **Fast Storage (SSD/NVMe)**:
LZ4’s decompression speed offsets its larger block sizes.
- **CPU-Bound Workloads**:
If your system has limited CPU resources, LZ4’s efficiency reduces contention.
- **Frequent Random Access**:
Even with slightly larger blocks, LZ4’s near-instant decompression minimizes latency for scattered reads.

---

### **5. When to Prefer Zstd for SquashFS**
- **Slow Storage (HDD/Network)**:
Smaller blocks reduce I/O time, which dominates total latency.
- **High Compression Needs**:
If storage space is critical (e.g., embedded systems), Zstd’s better ratio justifies slower decompression.
- **Write-Once, Read-Many (WORM)**:
Archives read infrequently benefit more from Zstd’s space savings.

---

### **6. Practical Recommendations**
1. **Benchmark Your Setup**:
Test both algorithms with your hardware and data. Use:
```bash
mksquashfs [source] [output] -comp lz4 # For LZ4
mksquashfs [source] [output] -comp zstd # For Zstd
```
2. **Adjust Block Size**:
Smaller blocks (e.g., `-b 64K`) improve random access but may reduce compression ratio.
3. **Prioritize Use Case**:
- **Real-time systems (VMs, containers)**: LZ4.
- **Archival/backup on HDDs**: Zstd.

---

### **Conclusion**
For SquashFS on **modern SSDs or in-memory systems**, **LZ4 is typically better** due to its decompression speed, as I/O overhead is negligible. On **HDDs or bandwidth-constrained systems**, Zstd’s smaller blocks reduce latency. Always validate with your specific workload!

[rych]

my freshly reconverted version of Nemesis XFCE4, which uses LZ4 modules

Does your system feel any faster at all, honestly?

[roadie]

my freshly reconverted version of Nemesis XFCE4, which uses LZ4 modules

Does your system feel any faster at all, honestly?

@rych
For me, LZ4 is much faster on a machine with SSD and Porteus v5.1. I don't use USB installs, or USB anything, so can't speak to that. Boot time is cut by 4 seconds and response time of a browser like Librewolf is nearly half when first opening after boot. Subsequent openings are basically instant with LZ4 compression whereas it's about 3 seconds with ZSTD.

That's about as technical as I get regarding tests, though I did do some tests with compressing and decompressing with ZSTD and LZ4. I found LZ4 was hands down the winner.

@M. Eerie
Thanks, very nice find.

I'll be staying with LZ4 compression, I can live with the modules being larger. Something I found strange, (and rather disappointing) is that I converted my Porteus v5.01 install to LZ4 compression expecting the boot time to improve and it made very little difference, barely 1 second. Being disgusted, I didn't test further (chasing boot times tends to cause one to pull hissy fits when not getting desired results). I may repeat the test, though I seldom boot v5.01 anymore and now with LZ4, v5.1 matches it.

[rych]

LZ4 is much faster on a machine with SSD and Porteus v5.1

OK then, when I upgrade to Porteus v5.1 that's when I also convert to LZ4, unless ...

[M. Eerie]

Does your system feel any faster at all, honestly?

Sorry for the late response, but I've been offline.

I think it takes much (much!) longer to explain it than to repack your modules. You can trust me, the I/O operations are instantaneous. The process, I’d say, is as well.

And that's using a USB drive. So if you use an SSD...

In my opinion, it’s the fastest Porteus experience.

As I mentioned at the beginning, it makes you feel like it's a "real system." What would be truly useful is the impression of someone using an older computer with fewer resources. Mine has 32GB of RAM.

[ncmprhnsbl]

i'll be adding this as an option in the various module building related scripts in upcoming porteus and nemesis updates.