Watching/notifications

[rektide]

A huge huge huge part of file-system management & capabilities as we use them is being notified about when something changes. Without this, awful awful awful terrible badly performing highly costly & deeply inadequate hacks grow up like weeds, all over the place, as bad terrible coders do an awful job of probing around to figure out "what has changed?". Huge amounts of engineering effort have gone into trying to tackle this issue. Works like node-watch (attempting to fill the painful dx gaps in node's fs.watch being non recursive) & watchman grow like weeds, consuming developer year after developer year of time to make maintain & sustain.

I love the works proposed here, DEARLY, and think they are so on target & in the right direction. But if we don't ship in 1.0 some kind of reactivity, some kind of file watching, given what I see on the rest of the web, it will literally never ever happen, and we will be very very very sad with the years of strife suffering misery & terrible problems that were so easily avoided had anyone bothered to stake in on caring about a moderately successful go at what file system apis need to be & do.

Please please please please please. Please make native-file-system api have watching capabilities. Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes, let me know. Without this, the amount of polling & state checking will boil away the oceans, be the most tragic waste of our planet we could imagine. If this is included, we can blissfully begin to carefully slowly rebuild the web in a way that coherently integrates the user experience with the stored shared mutable state that is the filesystem. Being able to see & observe those mutations is a crucial capability that I would put much on the line (were I aware of any such means to) to bring forth to the web, to native-file-system, as it turns this from a dumb, dead, passive tool into something involved, dynamic, & reactive.

Please kindly include some kind of fs-watch like capability.

[pwnall]

Thank you for your feedback! Glad you care about this API! We also think it's a very important capability for the Web.

Watching isn't going to make it for 1.0. Implementing it would take a lot of work for a browser, so holding back 1.0 for watching would delay all other use cases that can benefit from what we have today. I understand that watching is very important to you, and it has some applications that I care about, but I wouldn't be fair to ask everyone else to wait while this happens.

If you're interested in making this happen, you could help by putting together a Chrome CL that shows how to implement watching well on Windows, macOS, and Linux. This would help assess how difficult it would be to implement this in a cross-platform browser -- my intuition is that it'd take ~2 engineering years, and I'd really like to be proven wrong.

[guest271314]

Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes,

Does that definition of "changes" mean appending or removing a single character at, that is, any .txt file outside of Native File System methods being used to do so; for example a text editor changes a file that Native File System has permissions to read/write? Would changes to permission of the file performed at shell also be considered a change; e.g., a file permission being changed to read-only and/or write/execute during permission for Native File System being granted?

[rektide]

@pwnall The only thing that turns up when I search for "chrome cl" is this link to something in ChromiumOS, and there's not enough context on that page for me to understand enough to try to follow along. I don't see anything in the Developer Guide that seems very promising. I was thinking perhaps (https://chromium.googlesource.com/chromiumos/docs/+/master/developer_guide.md#Making-changes-to-packages-whose-source-code-is-checked-into-Chromium-OS-git-repositories) was going to give me something to chew on, but it doesn't seem to match up with anything in I see on the "create a cl" page? What was your ask again?

As for your ask for a cross-platform PoC, I have no access to Windows or Mac systems. I've also never worked on Chrome before. It probably would take me two man years to do, sounds about right.

I do want to help but this is also terrifying, and feels a little bit like being sent to never never land.

[rektide]

@guest271314 yes to all of the above. I am a bit of a file watching maximalist.

The filesystem is our shared source of truth in computing, and being able to observe the truth happening is important.

[guest271314]

@rektide A "a non-terminating procedure that happens to refer to itself" pattern which reads the file continuously? A copy of one or more files would need to be stored, correct? Consider the case of same file open in several tabs in a text editor, or multiple text editors referencing the same file; how to determine which file is the original? Timestamps, or is the original file not important, only indexes of the number of files open having the same path and name and differences between the files? How would the developer be able to process all changes to files in an event-based or Promise based pattern in such a case, that is, the same file potentially being changed multiple times based on input from multiple programs accessing the file? Have not tried reading read-only files in some time; what should occur when a file permission is set to read-only and permission to read the file metadata is requested?

[guest271314]

@rektide A DirectoryObserver and FileObserver, similar to PerformanceObserver, IntersectionObserver, and MutationObserver?

[guest271314]

Since the user must know the directories and files which potentially will be changed by a write procedure, the program can create a ReadableStream to read every byte of an instance of let stream = writer . asWritableStream(), or similar directory and file writing stream. During the write, notify the changes made to a copy of the original file, then delete the copy, the changed file taking the place of the original file reference.

Code to implement checking for changes made during writing to an existing file and notification code is not an issue.

Notification of changes to permissions of files could begin be creating a time range in which the notification should occur, to avoid a "a non-terminating procedure that happens to refer to itself" pattern relevant to asynchronous code running continuously, consuming computational resources, even when no permissions are being changed during every call to the procedure. For example, from now until then check if directory and/or file permissions changed. Or, a method that checks only when called, not continuously.

How many times should a notification occur? Once, when single instance of write/read procedure completes? Or should notification occur once for each and every change to underlying data source; and directory and file permissions?

[guest271314]

See

• https://stackoverflow.com/a/880278
• https://superuser.com/a/634313
• https://github.com/tartley/rerun2

[jespertheend]

Being able to listen for changes is an absolute necessity when making something like IDEs, photo/video editors or any web app that uses its own type of file browser. I'd love to see this added as well.

Most OSes include their own apis for watching for file changes, ReadDirectoryChangesW on windows for instance, or the File System Events API on MacOS.

I'm not too familiar with browser development so I can't really comment on how long it would take to implement something. But by having these apis to our disposal hopefully it should take a lot less time to implement this than when browser vendors have to implement their own file watching system.

[davidbarratt]

someone asked me if I could make https://github.com/onedrivejs/onedrive work in the web browser (i.e. no need do download or install anything at all), and you know what, it almost could, but what's missing is being able to watch a folder for changes, which is required for a syncing app like this one to work.

[Eloque]

someone asked me if I could make https://github.com/onedrivejs/onedrive work in the web browser (i.e. no need do download or install anything at all), and you know what, it almost could, but what's missing is being able to watch a folder for changes, which is required for a syncing app like this one to work.

I've managed to make a very simple monitoring solution by maintaining a list of files locally and then checking that against what I get back from the directory listing. Works pretty well. I run into not being able to store file handles or references. So, every time I start the page, I need to manually select the folder to monitor again. I am fine with having to ask for new permissons, but selecting it again and again is annoying.

[Goctionni]

I've been working on an app recently and the only reason I had to make it an electron app rather than a native web app, is the lack of a file system api (that includes watching). A watch functionality would be a really big deal, it would make native web-apps so much more powerful.