Changes:EXIT xzm

[dreadbird]

New changes

_uid_ is the default for changes mode

logSPRF incr or full can log the changes from initial creation or just create new

a marker file is placed in /dev/shm and if there its still the same session. rsync dryrun is used to check what files need to be copied to the next .xzm.

for the python version it will take me about a month to fully transfer everything over as im targeting this to work with linux windows

with the introduction of .toml files an icon will be in the start menu to open the config directory to change settings
a setting desktopicon will put an icon on the desktop ect

for nemesis a setting create startmenu will insert an icon so it doesnt conflict with the module active/deactivate.

Work sites
add rules for preferred or targeted directories to hash or watch by checking at shorter interval

The index will be refreshed twice per day or desired setting with a button to refresh if the system is changed.

All new files are captured and encrypted in a .gpg file with Go subprocess. This is when the app isnt running. And turns off after one hour.
If the gui is open the db is already decrypted and can be directly inserted.

When opening the gui a check for one or more of the new .gpg files is made.

If a directory in the index is empty an entry empty is made that way you can tell if files have been inserted.


If a file cant be accessed due to permissions an entry known inserted so this can be shown as mapped there is a file here.

What is in the index table:

// Index (from base modules 000-003)
//
// linux -executable
// bin etc sbin usr opt/porteus-scripts
//
// /lib /lib64 /usr/lib /usr/lib64 var/lib *.so*
//
// all of /etc /home /root
//
// may add more .config files ect

// windows - executable
// .exe .msi .bat .dll .vbs .js .jse .ps1 .reg .lnk
//
// .vhd .vmdk
//
//
// .ini conf .cfg .sqlite .db

for windows NTFS drives have an MFT table which can be used and is much faster than doing a directory tree walk
This will require the use of C# and or a library to parse it if there is one available

I currently have Go concurrency walking from a 5/5 split or 6/5 split from front to back. So if I can use the MFT it would be much faster

I will put both methods in then if one breaks or doesnt work there is a fall back. The speed shouldnt be an issue anyway but would be nice to do it the fastest method.


The use would be open it when diagnosing what a new installed application does. An output of files created can be made from an install.

Also a review of a search for new files can tell what files the application uses.

So it works like before but will hash the index to capture event where the same mtime but different checksum. The worksites are hashed more frequently for this.
on porteus this isnt a problem because even if the same mtime its put in the changes folder.
Then there is protection with every new search to hash all modified files.


A search for a filename can be made from the logs table because of the fsnotify inserting new files and or the MFT on windows. if it is not found it can then make the search like normal.

So if it only scans for created files it wont get overloaded by events. Also exclude from a slice of known directories such as mozilla/.mozilla/.cache

This will alleviate the problem of apps trying to do too much. If you want to see registry changes ect use a combination of apps ect.

Summary of methods:
1. Same as before but using arrays with multiprocessing ha. Like the system table the index is checked and feedback if something is changed.
2. New qt gui for diagnosing questionable apps and setting rules for index and worksites. The app can be left in the tray to keep the Go process running or closed and will run for one hour.
index is updated twice per day / by button
the worksites are hashed at interval / by button
this will capture same mtime

For porteus there will be
save-changesnew
save-changesnew python tk same as above but with arrays
save-changesnew python qt has index and worksites. searches can be made by command line or button for searches ( inverse? )

windows
save-changesnew python qt




I will release the array version in a couple weeks and then the qt version would be for windows and or diagnosing in porteus. The size would be a lot larger because of the qt packages ect. So the array version would be preferable for linux for the simplicity. A custom picture can be loaded or use default

[dreadbird]

Found a small bug in the python ha from optimizing stat call.
Updated .xzm logging logic for _uid_
Added no compression for pydb on size limit
Worked on inotify logic tested working
See release note on .xzm saving

I found out that I can use powershell to walk the directory tree in windows with parallel and insert directly into a database with modified time path changed time and filesize using windows api
there is also WSL2 which can use the find command on /mnt/c . so python can check if WSL2 is installed and use that

I also have a MFT parser so if you want to find a file or for example a .pdf it will be able to parse the MFT and return all .pdf files. It can also return a search time of all files so its very fast.
NTFSwalker.exe can extract a $MFT from a live partition and then insert it into python. so there will be that capability on ntfs volumes.

xRC works but it only captures the directories at the time of starting inotify. So it while it works its not ideal and have to use python watchdog to build a dynamic watcher.

So in the next week or so I can launch the array version and then the qt version with windows support after that.


Windows qt will use a Go binary/fsnotify with two workers with their own writers to two seperate ndjson files while the app is open or in the task bar. if a search is made for a filename it will check the json and if the file is found return the path. Otherwise search as usual. This will use 4 cores and search recursively in a split of directories so its fairly quick. Will integrate the windows index to cover those directories.

So will leave it at qt for windows with the python binary. The problem with doing too much is you work away from the design principle and things can break.
The good thing about polling with file searches is its efficient and can build off of this by proving stability. Looking at file watchers and things can go wrong fairly quick and consume resources. So even with Go im not liking it too much

near completion just stress testing it and its a full conversion just have to make sure everything is balanced the size is at 49kb

Update
Got everything tuned. I have decided not to release qt for linux as it adds unnecessary complexity. With windows you can parse the mft and its not command line friendly so gui works also the python binary works.
So if I leave it as is so it can be modified ect and if qt is released at a later date it would be tested proven as well. Looking to release it today with the new .toml config file which will be in the startmenu.

[dreadbird]

python developer buddy

full multiprocessing proteus shield, hardlinks, hashing and hybrid analysis
array streaming from find command
new config file
much easier to update across distributions
re:
typo fix in detectcopy func py editions and verified proper unescaping in all versions. Will look at just skipping files with weird character or embedded newlines ect.



bash/Python backends

save-changesnew Porteus 5.01 5.1 10/07/2025 v1.2.1
https://drive.google.com/file/d/1G73AH3 ... drive_link
Sha256: 6c6df87bb9090c4658575eba32258e13fecc051ac53cc6229170cf8709f2b42e




save-changesnewNMS Nemesis 10/07/2025 1.2.1
https://drive.google.com/file/d/1SydrB5 ... drive_link
Sha256: 539912d8271e6141b18926cde39dab22b9112f5c64b6098f75527a151f99f5d2




Python edition
save-changesnew NMS py 10/07/2025
https://drive.google.com/file/d/1ClmrZ7 ... sp=sharing
sha1 6c7417d356788640c9379908da65a62b6fba8133



save-changesnew Porteus 5.01 5.1 py 10/07/2025
https://drive.google.com/file/d/1O0i-Cy ... sp=sharing
sha1 ecf62a6f9319dcb50fb16e0081bf36b1d711d2f9




prev hashes

NMS: 891b8adb9bdd998d2376b216b53db2e127c2ad798481e5ec3d78045dcddd6c54
port: 268ec2d7775c7c40744f91e765748b1cc242279984763c1802d1fba055ba5a84
NMSpy: 9a35a8670297d1dcf1b12c619f3aadad0e79fea7
py: 5fd1210d3f7a69706a7336b5606f76fa9a0ba664


chmod 600 -R /root/.gnugpg later on if you get message about permissions
porteus 5.01 steps
install gpg2 for now as it has gpg-agent pass caching



tk for database viewing
https://drive.google.com/file/d/1IfC8x5 ... sp=sharing




currently working on the windows edition and this will allow me to vet the code and then I can set aside a day to check everything but the core is there. So will update as soon as I spot something but it should be stable


im just going to stop here and go back and check everything on the linux side

Updated linux side and finished the windows core so will work on the qt gui and look to have it done this week

Template for qt gui




Depending how clunky it is with the display results Ill just fill it with a distro / os logo with indent so it fills the space. An indent into the ui would make it clean

First sheet from the powershell script with 8 cores to 1 database then to ha



Working on a couple graphics to take up the rest of the ui. so the qt version can have a stat page and hold info such as how many files have been processed ect. So thats what I will focus on is feedback and stats. Then down the road
can insert watchdog into qt ctime > mtime to bypass 1 of two loops for more efficiency ect.

I am able to generate an inode or unique file id with pywin32 and use that for windows. Instead of user owner and permissions there is user and mode. I can also use Domain for group but decided that wasnt worth it but can always add that in
Also documenting as I go so it can be ported to mac os


this was done using WSL1. it will set the default distro to wsl1 if it fails fall back to powershell. Have it complete just have to do the gui

Finished the mft using pandas to output quickly. I left in the ".xzm" logic and will look at having a drop down box can select .rar/.zip and file type. ie compress all .docx to .zip for time specified to desktop.
the qt app can get the mft or one can be imported with:
enterro FTK Imager
wiztree
kape

Summary of windows edition:
Search with WSL find command or powershell (default)
Scan the index for unwanted changes to the system
Find out what files the windows system uses from database by count
Find a file or file(s) by extension

Output the mft and also with a mft difference file. Supports importing a mft
This was found to be the best method. dropped from default as its more of a diagnostics routine.

Lastly to find out what files an application installs instead of watchdog or Go will use recursion and indexing. log created files to ndjson files for find file. Then look in the ndjson and return the path. If it is not found start a search. So there is next to no overhead



For the new gui or Updates/changes and manual
https://docs.google.com/document/d/1EJA ... it?tab=t.0

[dreadbird]

Update on progress its looking really good. The interface is very quick and have both WSL and powershell in the main searches, system indexing and find file. mft was moved out and will be a different stand alone application. So once released the qt gui will be available for both windows and linux. Scalable custom image, a nice terminal display with minimal output and is lightweight/scalable. It works both with the gui and commandline. So again the core so this will be 50mb with low overhead and then there is tk which is 1mb. so will focus on stability which is why mft had to be moved out far too many moving parts

down to the last part which is a custom indexer takes all the directories changed time and caches those to a gpg file. When refreshing the index it can just jump over directories that havent changed. To start it takes all the root directories in C:\ then randomizes them and split them to differnt cores. Then those all go out and hash the targeted windows files (.exe .bat .ps1 ect). So it takes the search and increases the speed. The index is fully update to date. A scan would have to scan all the files but would be fairly quick as its targeting certain files

I also am going to put in mft. the problem is windows doesnt have a tmpfs or ram disk. it can be 1gb so its ok for verifying files after installing a questionable app. I just cant use it in my app because I dont want to be writing data to disk. The option will be there to grab the system mft or import an mft and output to desktop. Then you can also diff the mft file with the app. so you could diff it and see what files the windows system uses ect. It can use all python with rust hooks from omerbenamram/github and his parser. or MFTECmd. This way if windows ever changes it it doesnt break the app as Im not using mft in the main search.

Should be done in a few days. The app can also be used to store notes and encrypt to the gpg as there is enough interface left over ill put that in as well.

Here is the final template which I dialed in



The index instead of being refreshed you would just clear it and rebuild it. This was found to be the best. As you dont want things constantly changing once you have an index set. If there is a miss rate of 30% a message will be displayed to suggest rebuilding the index which takes about a minute



This was done with powershell. the delay is that it has to walk all the directories with 8 cores write it to a database then python grabs it and processes it performs hybrid analysis and then does all the encryption. So it shows how fast the find command is even using windows api in powershell its much slower. But for an entire hard drive its not bad especially if you can find a file


I have the mft feature with an option to select a ramdisk. It will do all work there if available. Then output the range exactly as rntfilesxMftchanges(Range).txt
if it is ran again there will be a difference file. xMftFromLastSearch(Range). So you can see what an installed application changes or what files windows uses.

Instead of listing all the information it is listed by modified time filepath. so same format as all the searches for maximum clarity.

Summary:
1. The search lists the modified files by cutoff. If a previous search was done the difference file will have the new files since the mft last search.
2. Requires only 1 binary. This is to dump the mft with icat or ntfstools which are both integrated. This method uses omerbenamram/github python rust mft parser then to pandas.
3. If MFTECmd is installed it will use that to parse the mft. Then to panda dataframe for comparison.
4. You can produce a full .csv from an imported raw mft (MFTECmd). Export the system mft to raw format (icat/ntfstools).

[dreadbird]

Customized to produce real time feedback of progress to the progress bar including from multicore searches. This is done without effecting performance or unnecessary wait times to loop over the count. With use of an array to produce the steps before entering key loops ect.

Main multicore areas
1. Main search multiprocessing pool map
2. pstsrg ha multiprocessing pool executor
3. build system index ..
4. scan system index ..

Feedback from Mft features and powershell also while searching or running


Currently stress testing
Building the filter for windows filter.py
Final Calibrate from a mft ( is the find command and powershell reaching all directories?). Also the indexer dirwalker.py is that reaching all directories?

Update:
Currently testing and setting progress bars and things but should be completed in a few days. With the idea in mind to easily convert it to linux less the search options and customize it for linux. The terminal would launch in linux with the gui. For Windows its just the gui but with both methods supported.



Image for the system index. on linux its called proteus shield so I kept the same name for windows. This will be the logo for it.

Made a porteus crest with inkscape and hopefully a brother will pull through for a few more images.
https://i.imgur.com/XsoZ2WH.png

Moving into testing. Its taken me a while to get things going as had to discover the best way to proceed from all the options. The core principle is to use tools available and build a foundation to allow for customizing. What setting you decide to use the app will follow for instance if you choose ram disk it will fill the output box to output the ramdisk as well. Otherwise it will just do the work on the ramdisk. If you dont have a ram disk no problem because the mft thing would be an occasional use anyway so a write isnt a problem. I do have streaming but I moved that to a seperate script as its not necessary to have that built in. The more features added the risk of breakage and maintenance goes up.

The mft thing was a good idea because it is a reference point of the reach of the find command and powershell search feature. Both are at 100% and also mft replaces the xRC option because it can do all that with its transparency. So the app is 100% polling.

What is in the Mft difference file?

Unique entries of the new Mft if the times overlap
or
The old Mft search

ModifiedDate Time Filepath

Created files
ModifiedDate Time Filepath


If a search is made without an old search there will still be a diff file with the created files.

Created files are cross referenced from the usn jrnl and mft data frame. The usn jrnl can be read with fsutil in powershell so I put this in

[dreadbird]

This was done with WSL and the find command multiprocessing pool for 65000 files.

// windows - executable
// .exe .msi .bat .dll .vbs .js .jse .ps1 .reg .lnk
//
// .vhd .vmdk
//
//
// .ini conf .cfg .sqlite .db



With os.scandir recursion multiprocessing executor. The find command being faster due to the way its split (first finds the files and then splits). With os.scandir one or two processes end up with the bulk. as the base directories are randomized and sent to different workers. So can change it to scan all the files first and then send it for multiprocessing. The overhead would would be less but I think its acceptable as getting too fine grained can lead to maintainability problems ect



os.walk to first find the files and then send to multiprocessing. Tweaked and finalized on the right. so maxed out. I tested with powershell and it would be slower than the find command although it returns the files in 22 seconds ( find command is 19). But then it still has to convert all the meta and print it to stdout or to a file. Im seeing if I can write it to a database and do it in one step



Powershell doing all the hashing, inode, hardlinks ect to 8 databases then to 1 database.

so am now going through all the features and making sure they meet standards. Then have to put all the licenses in. Even though im not shipping the app with the mft stuff. The searches are verified by mft and signed in the docs when the test was done.


Summary:
dirwalker.py for system index building and scanning




Notice the distribution from the systimecache.gpg which is all the directories on the system. the bulk of the work is in program files. I was able to do round robin distributing and got to index and full directory caching of the system in under a minute with multiprocessing

Update:
I am going to put in a feature Downloads which is a push button. If the system has been indexed recently it can take the directory cache only search directories with added files and return the results in a notepad popup. If the system hasnt been index recently it just popups that the index is out of date. The editor can be selected in the config.toml.

This would be a rapid way to find any created without parsing the Mft or doing unecessary work and having watchdog or fsnotify ect. As the search is made any new directories are added to the cache. All directory modified times (Only when files are added removed or delete) are updated. So the next time its used its fully indexed. This however does not change the system index as that is not a good idea because once an index is set it has to remain. Then if changes are made to the system reindex.



The download button refreshes the cache and finds all downloaded or created files very quickly. the total time includes updating the cache_s table


What my initial plan was to reindex the system automatically twice per day. and have work sites where you can select folders that are scanned more often. This is to catch if a file was modified with a fake or the old modified time. So when everything is set I can easily add that feature in but however this can be a major problem with apps. they jam a bunch of features in and even differ from standard methods. So between two apps you similar features that may or may not do the same thing leading to a massive learning curve.

So overall have it finalized from testing different options and finding what works and what doesnt


Final layout


Note: Caching can be disabled in the config.toml. With caching the download button is available. As well there is another table cache_s which will give a breakdown of the most active directories from most recent to oldest. with file count, index file count and max depth of the system

[dreadbird]

Made progress and have it bundled with PSSQLite/sql for powershell and gpg. If a system gpg4win is installed it will use that otherwise it will use the /gpg/ from the app.

I modified the mft parser (eric zimmermans mftecmd) to output directly into memory as stand alone .net 9. filesize is 75Mb. by outputting to stdout instead of a file

if you just want to use the original one it also accepts that just goes to a file then processes from there. filesize is 3mb

Methods supported:

icat/fstat or ntfstools > mft raw > mftecmd > mft csv > memory csv
icat/fstat or ntfstools > mft raw> python rust hooks(omerbenamram) > mft csv > memory csv

mftecmd(.net 9) > memory csv
mftecmd > mft csv > memory csv

Summary (required 1):
icat/fstat
ntfstools
mftecmd

python hooks only needs one of the above. if mftecmd is installed it uses that instead of python hooks.

So the app will ship with 2 versions. .Net 9 or regular. for different filesizes between the two.



I also made a custom script for calibrating that adds a cutofftime to the csv parser. This will search the mft and output the results in a .txt file. The range back is set in the script default 24hrs. This script takes 4 arguments action. modified, created or deleted or deleteddirs. and returns

Mft search by action for cutoffdate
DateTime FilePath

I have two one for mftf.exe by ignacioj github. His parser is very fast and is only 56kb. It can only be used for new or created files found out it focuses on one mode in use files or another in use files and directories. Deleted directory paths are incomplete.

one for mftecmd .net9. This is the one I will release and am using. This calibration script can do all four modes created modified deleted and deleted directory and output to a .txt file.


So far I have the config.toml to set file extensions for the sys profile. Also excluded directories which is just Windows folder by default.
color is default unix green on black but there is an option for white on black (wob).
textsize can be set, font can be set
abs path to text editor can be set for a fall back if its not in path

Summary:
Taking steps to keep everything system specific ie C:\
Adding add remove index of different drives for downloads button and cache table overview on the ui. The main search and proteus shield can be switched from config option basedir. the profile is saved (extensions criteria) specific to the drive.
Adding compress button to zip based on find file search and cutoff time with support for 7zip and winrar.
When a search completes it opens the text editor if the setting is enabled
On app load loads last index selected via json file






nested color support

The directory cache or cache_s is used to find created files. a HDD will take about a 40 seconds to a minute where as a ssd will take a few seconds to make the cache file. But if its removed and indexed again its cached by the os so even a hdd takes a few seconds. Doing a search updates the cache fully and also takes few a seconds. So with caching the directories the idea being its focused on its strength and thats directory mtime. A full index of the files can quickly defeat the purpose of having an index the size would be too big and it would be slow and just doing a regular search would be faster in most cases. So this should be a useful tool for finding new files.

When a system profile is made the cache file is put in the database so theres a record of directories for the profile. A drive find file search will use this cache and create a new table that updates away from the profile and updates the .gpg so the table and cache file are in sync. If there is no system profile for the drive it creates a cache file. So its a hybrid approach and effectively saves a lot of time.

I added launch Powershell and explorer which will open in c:\Users\{user}\AppData\Local\save-changesnew.

Update:
I have the database and everything complete. i noticed the little features such as POSTOP which is the file doctrine so im putting the small details in. the database is solid with the system table and 65000 files it can load and be viewable with no pauses. I also made sure that in the searches creation time if it is newer than modified time the modified time is changed to the creation time and the last modified field is put with the old modified time and casmod is set. This ensures every possible file is shown in the results.

With mftecmd it can stream with a cutoff date from the mft directly into memory so instead of 1gb its 40mb. So this should be really interesting to see it in action. I have also implemented it into multiple core for a calibrating feature. With the idea of not tying it directly into the main search as its find command or powershell. The mft features use mftecmd exclusively or python dump hooks.


Started testing from the command line and working through all the features so everything is squared up with comments stripped. The scope is fairly big so starting small so I dont get overwhelmed. always debug manually and by using it so when its done its fully tested. So its taking a lot more time than I thought it would.

Adding things like super impose is slowing the release down but ide rather it be right the first time so Im going through each part of the scripts the powershell scripts and making sure its finalized before releasing

[dreadbird]

I was going to put a windows logo but I wasnt sure if its allowed so I decided to modify the main logo of the app.

crests

recentchanges brand
https://i.imgur.com/XlprXsb.png

dragon
https://i.imgur.com/xK10DuX.png

porteus
https://i.imgur.com/XsoZ2WH.png

tux
https://i.imgur.com/UjpBH3h.png

A custom one can be loaded with the custom button. A button to create the crest will convert a .png image it is required to be 250x250, 250x300 or close to there

icon


Superimpose


Any items from a system scan can pull their original record from sys_a and be combined the changes from sys_b for easy viewing

Adding super impose and proper formatting is slowing the release down but i would rather it be right the first time. So going through all the scripts powershell scripts and checking everything before release.

[dreadbird]

New ha logic will take all files with the same mtime and restat to check mtime to check that its not the same mtime with a different checksum. This proved to be the best method as a faked time stamp could have zeros or slightly off from the original and those are be caught before ha. With this change all of the remaing files (with same mtime) are checked and efficiently captured in the analysis. If a file changes during the search it can be detected and output in the terminal. So its useful in detecting cache items and is another part of the change.

As the time window of searches passes the ability to check for this no longer present. The scan profile feature which is implemented in the new Qt app as Scan IDX can check for this. Specific files for each drive can be added in .toml and are saved in
a .json file per drive by profile creation

Note: The changes in porteus is good for this as any changes including same mtime with modified content are shown there

Updated logic in ha fixed sys merging as it can pull from two tables more checks needed to be put in place. Updated

bash/Python

save-changesnew Porteus 5.01 5.1 v1.2.4 12/08/2025
https://drive.google.com/file/d/1G73AH3 ... sp=sharing
sha256: 5d54899373547c8a34895a4f927196c48e371cc841ca4a9bdbe74a5d439f3e08

save-changesnewNMS Nemesis v1.2.4
https://drive.google.com/file/d/1SydrB5 ... sp=sharing
Sha256: f6af6908ff5e83db12bd83ba971596ff2bf439c0777608aa5006617bb6f93d0b


Python edition
save-changesnew Porteus 5.01 5.1 py edition v1.2.4 python edition 12/08/2025
https://drive.google.com/file/d/1O0i-Cy ... sp=sharing
Sha1: 4bc427ac235787bc4bff64d0237181e833e84c23
save-changesnew NMS v1.2.4
https://drive.google.com/file/d/1ClmrZ7 ... sp=sharing
Sha1: 2bf525b58fd53614bfe6b6536a94dd7740bae9f9


tk for database viewing
https://drive.google.com/file/d/1IfC8x5 ... sp=sharing


previous hashes
save-changesnew Porteus 5.01 5.1 v1.2.3 11/28/2025
Sha256: c991224414db755528d8d02a4ecc26b77b2d950eb8c4cd0e13ab76493362ee64 ha correction
Sha256: 16891ac961d11c49f26061ee0dd5fc7b25f515a463537ec9f81249e92643efb9
save-changesnewNMS Nemesis
Sha256: e9b085dfb3604a24a6b6f6354f0d61475f8e2092f1eb1fa624cad3b464e6d72d
Sha256: 5c1acfab7d97b2c8e647c84fb39e915b9911128a83a55b867917612e4dc1efea
save-changesnew Porteus 5.01 5.1 py v1.2.3py 11/28/2025
Sha1: 4ac74f2a9c8429536e01ca4144191ece6f31d855
Sha1: 5c5d3447d8acbc7bb0cf9dd3fefce9318cbe2b66
save-changesnew NMS v1.2.3py
Sha1: ac07c3704cdc8625dc19f574b75ffd548d802ac2
Sha1: 1fd151b81ee4dc534ebd04dd84fac0080cb5e70a

for changes check out github or documents
github https://github.com/dreadwrr/Developer-buddy
manual https://docs.google.com/document/d/1EJA ... it?tab=t.0

[dreadbird]

I have everything complete just have to write a few test scripts to test certain sql functions to ensure they work. test scripts are needed because some of these run infrequently ect.
The biggest hurdle was reworking the filter for the pyinstaller as it cant run python scripts that are outside so I moved the filter, browser suppression and cache clear to a filter.toml. Escaping is different and that had to be solved.

This is because the typical windows machine wouldnt have python installed. a full python version without the .exe will be released as well

The application detects if the drive is a SSD or HDD. if it is unable to find out it runs a speed test and that easily tells the difference. A hdd is faster sequentially so before doing fsearch and hybrid analysis it checks what drive type it is from the config file. Powershell and find searches which are the main searches are serial as its much faster for that. The biggest speed increase is in fsearch and hybrid anaylsis for multicore and that is really fast for SSD. I also ensured its fast for HDD by switching to serial beforehand.

So aside from running a few test scripts and throwing data at it I will also check if it runs on powershell 5. Confirmed it works on powershell 5 that makes it so theres nothing needed for install. So I will put together a series of test scripts that I can run Then I can release it with confidence its up to standards

Update: Half way testing core features. There is a quick commands list in the COM menu bar which will list common gpg commands to copy paste. One is deleting a key it will list the key serial ect in the hudt. Another would be to decrypt a cache file to verify contents again just copy paste and the ctimecache.gpg can be decrypted. The text file can be edited to save commands for future reference. So in addition to this and saving encrypted notes it will be a good fit for linux. I will also make use of xRC which uses inotify wait to bypass the ctime loop. The first page will be minimal compared to the windows edition with focus on database

Testing postop scan IDX and file doctrine and then deciding on install forge or .msi if needed.

So hopefully release in a day or so but its looking really good

Final size of the pyinstaller .exe is 67Mb

mftecmd .net9 - streams mft into memory so no disk writes. this feature only applies to mft search on bottom right
.exe with .net 9 already installed 4mb
standalone .exe .net9 39mb

it works with mftecmd standard and that file will be 500mb as it can create a .csv from C:\
it will ship with ntfstools and because it has python hooks it can read the raw mft from ntfstools. That file is 1gb. so a ramdisk can be used if wanted
also works with icat and fsstat from the sleuthkit for dumping the raw file
ships with 7zip. will use system 7zip or winrar if it can find it for find file compress

testing system gpg currently which was tested before but making sure it uses that if its installed and then fall back to app gpg. verifying logging logs fully as that helps to quickly check if anything was missed.
and gpg tested so have it complete and will release on friday to find any last changes and package. I setup a venv and have the python requirements ready as well


A popup to specify time range and in this case 0 means to compress all results
Example using WSL. prune is dynamically built from exclusion folders from config.toml. the filter is also applied from filter.toml using regex. the result is a compressed archive that searches the drive
it is using the downloads setting and if one isnt specified goes to Desktop or AppData via combo box

The image on the right it can scan the entire drive output any new files by using the cache index. When its done the cache is fully updated
An alternative method to find new files using mft search. Then sometime later another mftsearch and the differnce file xMftchangesDiffFromLastSearch has all the modified files between the two

identified areas of improvement for 1.2.3 versions and applying fixes before releasing the qt version. Its taking longer than first planned. had to learn how to use pyinstaller install forge which install forge is a good candidate. advanced installer is impressive and can use .msi. so have those setup and ready to go. I will give it a few days to make sure everything is fully good to go and then release

windows 10 / 11 file search app
https://github.com/dreadwrr/Recentchanges Python
installation:
cd into directory pip install -r requirements.txt
python main.py

Windows setup (pyinstaller)
https://docs.google.com/document/d/1EJA ... 8qv65gr0wc

I was able to sign it with azure trusted signing and used pyinstaller as it is organized well. nuitka has better performance so if it was a big application like doing heavy work nuitka would be better suited but the file size twice as big

[dreadbird]

Moving over to linux.

The windows version windows subsystem of linux feature has the xRC capability built in which can be tied in later.
in linux xRC uses INotify wait which can build the tout file which are all files with new changed time. Result is being able to bypass one of two loops. so this feature would add optimization it with next to no overhead. xRC detects a downloaded file or a copy because of the preserved metadata where ctime is greater than mtime or casmod in the database.

Im looking at keeping tkinter for linux as the filesize is only 1MB. if I can streamline some buttons put in the encrypted notes and quick commands features in there.

I have optimizations for the hybrid analysis im currently testing and will be rolling out. Effectively using the cache which goes from .gpg to memory. The result is a very effective preliminary scan which can detect file movement during ha.

After looking at tkinter that will be for the bash / bash python edition and changes made to make it a cleaner ui. Qt will be brought to linux with the protection of being able to scan the idx. So going to optimize the ha and start converting it to linux



The image on the right is xrc. it is set to watch changes for 1 hour. so it preprocesses the ctime loop by running the sblk hashing the file. these are downloads or copies. so the entire search is halfed as only the mtime loop runs.

Have the preliminary port done seeing if I can put some tweaks in for linux and reshape the ui

I have it using polkit so it only runs the main search as root after password prompt. Gpg also pops up with a gui it was difficult to figure out but root and guest can share the same key pair.
I will include full root mode as a seperate launcher as it’s a live distro and so some may prefer that especially if having issues with polkit.

currently im working on the bash and will be releasing nemesis and porteus with xRC and all optimizations and tk. as its a base of the application its a good starting point. Nemesis is almost done so im just putting in all the error checks ect

[dreadbird]

I updated bash / bash python to be able to generate the sys profile in tk. also added a refresh button. Im looking over save-changesnew to see if I can implement any changes. then will release those for nemesis and porteus shortly
The hardlink button will reset the hardlinks and cycle through all log files and update the hardlinks so as search times drift you can update hardlinks automatically

As well when doing a search during hybrid analysis if a file is a system file it is added to the sys table (with timestamp, filename, changetime as keys) and the count for that filename is incremented. This will allow to see what files the system uses ect.

I updated the file doctrine and parsing as well as save-changesnew just going to run through everything and release for nemesis.

New features:
namingPRF alpha or numeric for randomized labeling of the modules. the alpha is nice but the random number is also kind of neat.
Implemented ap40 encoding. which just speeds up parsing in the loops. The reason for this is I put the filename in quotes and not on the end of the log for readability. the python uses this as well converting from bash.
went through and optimized the multiprocessing.
xRC dynamically half the search speed so optimizes further
mergemodulesod or overdrive. this is a new way to merge the modules without using mounts that I made it just merges the modules on the drive and not in /tmp. So will put a symlink in /extramod for that.
I optimized the dynamic saving of changes with /dev/shm with logic for edge cases to ensure the smallest module size is made. This is in place now already but I put in more logic.

Its taking me longer than I thought because everything has to be checked and tested again after making changes. So im currently checking Graphics and Graphics changes to make sure its all good



How it works is every file is labeled from the .xzms. Then the ones to be hashed are selected and a difference file holds all files not to be hashed and inserts those as well. So every file is in the sys table and allows for an accurate count of every file for the install

i ran the test scripts on hash collision checks in both bash and python and finished the check for the sys table update in hybrid analysis. So have the optimizations completed and tested. Im going to cycle the modes and check the documentation at the top of the scripts is accurate before release to nemesis and the port to porteus should be fairly straight forward

[dreadbird]

bash and bash python

Changes: save-changesnewNMS fixed typo found when converting porteus
01/17/2026 Updated inotifywait

save-changesnewNMS Nemesis v1.2.6 01/17/2026
https://drive.google.com/file/d/1SydrB5 ... sp=sharing
sha256: f2b7092af9c728f0123699ff6cc43259291c7724dd9879891134bf3d7476d9a8

Updated inotifywait
Updated xRC inotify script
Updated with py version changes and worked on save-changesnew
xRC tested and working correctly implemented from py version. default is watch changes folder. The cache is also updated when prebuilding the file_creation log. This if turned on will bypass the ctime loop increasing efficiency
Implemented hash verification recursion to ensure all checksums are verified this is because cache items frequently bounce around
Found flaw in collision detection with symlinks set sql to where null
save-changesnew new feature mrgAEX which can apply or not apply exclusions after auto merging. default is just to apply any exclusions after the merge because exclusions could have changed to save space
checked that all scripts use tmpfs for extracting and merge


save-changesnew Porteus 5.01 5.1 v1.2.6
https://drive.google.com/file/d/1G73AH3 ... sp=sharing
sha256: 664639bc3a9573b4ca9470bbd142148b7ecc12074b3139388087f42d66284e24

Updated inotifywait
Updated xRC inotify script
inotify from package manager to use xrc



the rsync version is different from nemesis but have it converted over using porteus alpha2. So after system start roughly 600mb and save again and 40mb. Using rsync dry run can make the modules smaller by using the shim put in on first system save. It shows the extract time so what it will do is extract all modules make since the first shim then dry run it and get the difference from that output. Then only save the exact files needed or have changed

notes for installing
tk to view database search for tk in package manager or from here tk-8.6.16-1-x86_64.pkg.xzm https://drive.google.com/file/d/1IfC8x5 ... sp=sharing
porteus needs pinentry dont use package manager for pinentry although it probably will work with 5.01. instead go to pkgs.org and download pinentry for current (porteus alpha 1 and 2).
sudo su
cd /root
echo use-agent >> .gnupg/gpg.conf
echo pinentry-program /usr/bin/pinentry-curses >> .gnupg/gpg-agent.conf
or
echo pinentry-program /usr/bin/pinentry-tty >> .gnupg/gpg-agent.conf
gpgconf --kill gpg-agent

to verify the agent
optionally start the daemon
gpg-agent --daemon
gpg --quick-generate-key "Test User <test@example.com>" default default never

should popup with curses or the tty.
if it fails try to run /usr/bin/pinentry-curses it will show whats missing usually libbsaun or npth as some pinentry packages dont have these
even then its important that the right pinentry package is used as pinentry-1.2.0 will appear to be working for alpha 1 but will fail silently. ensure current is used for alpha 1, 2 ect

also for a windows popup you can
For nemesis
sudo su
cd /root
echo use-agent >> .gnupg/gpg.conf
echo pinentry-program /usr/bin/pinentry-gtk >> .gnupg/gpg-agent.conf
gpgconf --kill gpg-agent

Porteus
same but use
echo pinentry-program /usr/bin/pinentry-gtk-2 >> .gnupg/gpg-agent.conf

if that fails try gtk+2-2.24.33-x86_64-5.xzm package.

previous hash
save-changesnew Porteus 5.01 5.1 v1.2.6 Currently updating inotifywait script update pending
sha256: 03cd2c209be0bbf3b43e2a04c9cc8e91c743398400a9ef2e42ed32351137c62b
save-changesnewNMS Nemesis v1.2.6 01/13/2026
sha256: 1e3995bcceea9c1d4b706a892e89bd5a8081dfbbd60c8531c52e1ecde2fca694
save-changesnew Porteus 5.01 5.1 v1.2.5
sha256: 8e70ce6b9247dfc7b7c95cfcde8a151894066a821715d29ca06d59638b61d83c
save-changesnewNMS Nemesis v1.2.5
Sha256: a1471cc7d869034b7335c7bc187d96a99d88c405c2ee6496aa50381a7e3468d2
save-changesnewNMS fixed typo found when converting porteus
sha256: a07942030384296f00de5aac5d57fe3942e19c87bed513aa3482571b7591b4b2

[dreadbird]

Python edition

requirements
gpg agent is setup for both root and user. repeat above instructions for user in ~/.gnupg this is needed for query
nemesis i think has pinentry. requires tk-8.6.16-1-x86_64.pkg from package manager
if anything else required maybe 05-devel.xzm

save-changesnew NMSpy v1.2.6 01/17/2026
https://drive.google.com/file/d/1ClmrZ7 ... sp=sharing
sha256: d5ce3e6f7426fdb893ce8fa5ba321b49fe0276df5a5d5f4156505e2e0862763f

streams the find command
changed cache to use dictionary versioning in all versions. with the python version loading the .gpg to memory
xRC implemented in python. Adds cache from inotify. This feature bypasses the ctime loop to increase overall speed and efficiency
finalized ha with notification of microsecond all zero and load balance between fsearch and hanly to achieve the best it can do with minimal overhead.

implemented a .gpg verification and auto delete system to make handling keys easier with command : recentchanges query able to detect conflicts
command: recentchanges reset can be used to reset the keys

new bash launcher script with the switching done in python. with support for Wayland and other distros
configuration file updated with new options and the way the find command works and it runs off of exclusions in the config.toml which is from the windows Qt version.
py edition also has a full logging system to ensure proper operation and debugging
inotify script also has a debug logger which is off by default. any errors are still logged to /tmp

save-changesnew Porteus 5.01 5.1 py edition v1.2.6 01/17/2026
https://drive.google.com/file/d/1O0i-Cy ... sp=sharing
sha256: 986b3ac03cf6a00f30094189a6ab636498b3258e50fa5ac266a7b0e1a4df9f0b

inotify from package manager to use xrc
for porteus alpha 1 or 2 pinentry 1.3.2 current. can be found on pkgs.org. porteus 5.01 may work with pin entry 1.2.0 if having a problem its pinentry version mismatch

for porteus put in error handling specifically for pin entry and key detection. that and with automated key conflict and delete\reset process should be convenient then having to manually do it.


previous hash
save-changesnew NMSpy v1.2.6 01/15/2026
sha256: db73327bb5ff705cfdf93221e378433f1962b1d23263ba07053117058902a666
save-changesnew Porteus 5.01 5.1 py edition v1.2.6 01/17/2026
sha256: 419c29997861dfa4b72c68f3bbe4bb80726943310c12bcdd1e48baec614e30b0


updated both py versions and they all have the same .py files. added the extra logic found from the porteus version to nemesis. wrote hash_profile script for query and prompt for root password as query doesnt run as root.
fixed icon for porteus for config folder to showup in app menu

Will update the bash \ bash python inotify script tomorrow other than that all versions are completed. I accidently deleted the ui for the Qt app when testing the porteus version so I have to try to reverse it from the ui_mainwindow.py. Luckily I saved the ported qt files I did from windows. Will work on graphics for xRC and mrgAEX so next will work on ui and graphics always fun to use inkscape

Updated inotify wait in bash \ bash python versions. My usb stick was dying and had to make a new partition for porteus and accidently formatted the wrong drive. used photorec and testdisk and was able to recover the ui. had this not been the case there is no way to reverse ui_mainwindow.py to .ui.

Working on testing qt linux version and updating documentation.

[dreadbird]

reworked the ui for linux
The database streams from the tables so it doesnt slow down and results are displayed in realtime. Im surprised with libre office and modern csv that they crash so easily when sorting


this is the non xzm profile. I have the open file manager and terminal as root so it works with porteus but it may break on other distros. For example opening terminal as root isnt a big problem its the file manager that would be. So what I will do is make it so im not putting in things that might not work on other distros. that way it can be stable and customized later.

If I make those kind of changes I would have to boot into debian and others just to test it there

also in case polkit doesnt work it will prompt in the terminal. Testing is almost complete and then I can release the source code for making a binary with pyinstaller if wanted but the idea with linux is just release and use the python from the commandline



find downloads feature

uses the directory mtime cache and looks for new files only in dirs that have new mtimes. The cache is then updated for the next search

Update: ok so logging have it asynchronous using queues. I put the xzm profile from 1.2.6 for the shield in to a class that can be customized. So can either use specs in the config or the custom class. os.scandir is very fast and have it working with no refactoring. so by the time its filtered all multiprocessing has to do is hash and it works great.

just going through to make setup is fine like prompting in terminal if polkit is broken. theres all sorts of weird linux quirks but not as bad as windows which file locks

Scan profile
returns any files that have changed by checksum. scan profile or scan IDX was made to detect files with same mtime and different checksum
and results appended to most recent diff file if one doesnt exist uses a default diff file
if show diff is true scan profile can also show:
symlinks with new targets
all profile files that no longer exist
dirs that had no files when profile created but now do
new dirs since profile created

comparison between the find and fsearch to just python.

find fsearch: 13.058
repeat: 4.628

python scandir: 17.251
repeat: 3.041

Find fsearch
2.557 Collect files with find
8.046 multicore
repeat
1.969 repeat
1.152 repeat

python scandir
1.179 Collect files
12.307 multicore
repeat
0.421
1.142

the extra time for scandir is round robin chunk splitting which adds 4 seconds so it can efficiently hash the files.

summary calling find and parsing adds twice the time



I completed the drive mapping and came up with a solution for quickly changing sys profiles.

if a drive changes mounts ie from /mnt/sdc2 to /media/opt it will update the json, move the cache files and rename the database tables.
realistically at the most you might only ever use 2 sys profiles. this provides a convenient way to change the base drive or target.

when one drive is selected as the base the other indexes can be searched with the find downloads button.
the new base can perform scan and build IDX on its profile
by switching base drives you can search for files or compress all results with the option of filtering by time to a .tar or .zip.

with zipping if stripped paths is enabled and there is a duplicate the duplicates keep their absolute path so there is no conflicts



this is a first run on nemesis it has all files from the base xzms and is using the XzmProfile class built off of 1.2.6.
so 6000 files hashed and 40k remaining non-hashed. so when doing a file search it can pull from there and tell its a system item and also if it hasnt been updated in the time peroid (5 days) a warning is displayed.

I have it to the point now im stripping commenting and have it refactored. so its a good baseline as going in either direction can lead to failures or clutter making it harder to debug. here it can be modified and optimized