libinotify-kqueue is a library that implements inotify interface in terms of
kqueue/kevent API available on Mac OS and *BSD systems. The original kqueue
backend seems to be a predecessor version of the code that is currently present
in libinotify-kqueue. Under the hood the library implements a sophisticated
filesystem changes detection algorithm that is derived from the glib backend
code.
Updating the native glib kqueue backend requires substantial work, because code
bases have diverged greatly. Another approach is taken, instead. libinotify-kqueue
can serve as a drop-in replacement for Linux inotify API, thus allowing to
reuse the inotify backend code. The compatibility, however, comes at cost, since
the library has to emulate the inotify descriptor via an unix domain socket.
This means that delivering an event involves copying the data into the kernel
and then pulling it back.
The recent libinotify-kqueue release adds a new mode of operation called "direct".
In this mode the socket pipe is replaced with another kqueue that is used to
deliver events via a kevent(EVFILT_USER) call.
Employing the direct mode requires minor changes to the client code compared
to using plain inotify API, but in return it allows for reusing libinotify's
algorithms without a performance penalty. Luckily, all required changes are
consolidated in one file called inotify-kernel.c
This puts us in the best of possible worlds. On one hand we share a lot of code
with glib inotify backend, which is far more thoroughly tested and widely used.
On the other we support a range of non-Linux systems and consolidate the business
logic in one library. I plan to do the same trick for QFileSystemWatcher which
will give us the same behaviour between Gtk and Qt applications.
The glib test suite passes for both old kqueue backend and new libinotify-kqueue
one. However, the AppStream FileMonitor tests are failing with the old backend,
but pass with the new one, so this is still an observable improvement.
Relevant libinotify-kqueue PR: https://github.com/libinotify-kqueue/libinotify-kqueue/pull/19
The copyright from `git log gio/inotify/meson.build` is now included in
the file header. The following commits are too trivial to be
copyrightable:
- d10be6102f
- 03e86d000f
- 1741fc2c6e
- 8733d172a3
The file was contributed while the `COPYING` file for GLib was
LGPL-2.1-or-later, so was previously implicitly licensed as that.
Let’s make that explicit.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #1415
The license and copyright are already stated in human-readable form in
these files, so this should be uncontroversial.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #1415
This should not result in any functional changes, but will eventually
allow glib to be functional on CHERI-enabled systems such as Morello.
Helps: https://gitlab.gnome.org/GNOME/glib/-/issues/2842
Various gio modules include gmodule.h that requires the
gmodule-visibility.h to be already built.
To make this easier, just provide a dependency and use it where we are
building modules that do not depend on libgio_dep (that already includes
that).
Fixes: https://gitlab.gnome.org/GNOME/glib/-/issues/2982
The `inotify_init1()` API has supported this flag for a long time
(possibly since it was first introduced, although I haven’t bothered
doing the archaeology).
This saves a syscall when first connecting to inotify.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Since gmodule-visibility.h is now a custom target and produced at
buildtime, it might not always exist in time for use in other source
files. This was the case for gio-inotify.
Add it as an additional source file to ensure in-time generation.
When gio monitors a directory, it will delete the extra "/" at the end
of the directory string, but when the directory is "/", this will cause
the modified directory string to be empty, and eventually the
monitoring will fail. The solution is to delete only if the directory
string length is greater than 1.
Building GLib 2.75.0 on Linux adds various inotify-related internal
symbols to the ABI, which doesn't seem to have been intentional.
I went through the other libraries in the build system, and it looks
as though the BSD kqueue backend would have the same problem.
GNU symbol visibility probably doesn't do anything for gio/win32, but
for completeness I've set that to use hidden symbols too, on the basis
that it'll be easier to get this right if we're consistent.
Resolves: https://gitlab.gnome.org/GNOME/glib/-/issues/2811
Signed-off-by: Simon McVittie <smcv@collabora.com>
It must only be defined when building libgio. This requires some
workaround to allow include of some gio private headers.
When GIO_COMPILATION is not defined we cannot include individual gio
headers. We workaround that by defining __GIO_GIO_H_INSIDE__ in some
places. Also gdbusprivate.h is not an installed header, so it's fine to
include it directly.
gio/inotify/inotify-kernel.c: In function ‘ik_source_new’:
gio/inotify/inotify-kernel.c:377:3: error: missing initializer for field ‘finalize’ of ‘GSourceFuncs’ {aka ‘struct _GSourceFuncs’}
377 | };
| ^
In file included from glib/giochannel.h:33,
from glib/glib.h:54,
from gio/inotify/inotify-kernel.c:30:
glib/gmain.h:272:14: note: ‘finalize’ declared here
272 | void (*finalize) (GSource *source); /* Can be NULL */
| ^~~~~~~~
`-1` isn’t a valid member of the enum, so cast to `int` first. This
fixes a compiler warning on Android.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
So long, and thanks for everything. We’re a Meson-only shop now.
glib-2-58 will remain the last stable GLib release series which is
buildable using autotools.
We continue to install autoconf macros for autotools-using projects
which depend on GLib; they are stable API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Apparently this is needed for building PE libraries. It makes no
difference on Linux, where linking of the GLib symbols in the inotify
file monitor code is done lazily.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://gitlab.gnome.org/GNOME/glib/issues/1201
This gets the G_FILE_MONITOR_WATCH_HARD_LINKS flag to the state where it
doesn’t cause crashes, and essentially acts as a no-op. It will not yet
actually monitor for changes made via hard links.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://bugzilla.gnome.org/show_bug.cgi?id=755721
The call to _start() fills in the dirname, basename, and filename
arguments according to the following rules:
dir watches: dirname filled
file watches: dirname and basename filled
hardlink: filename filled
This doesn't map to how the current inotify backend works very nicely,
so we need to adjust things a bit when creating our "sub" objects.
https://bugzilla.gnome.org/show_bug.cgi?id=755721
If we can’t convert the inotify event mask into a GFileMonitorEvent enum
value, don’t propagate it to GLocalFileMonitor, since it hits an
assertion failure in that case.
This should no longer be possible since the previous commit to ignore
IN_Q_OVERFLOW events, but we might as well change this just in case
other bugs crop up in event mask handling.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://bugzilla.gnome.org/show_bug.cgi?id=776147
There’s not much we can do about them, and if they go unhandled, they
can propagate through to g_file_monitor_source_handle_event() and cause
assertion failures due to not mapping to a GFileMonitorEvent.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://bugzilla.gnome.org/show_bug.cgi?id=776147
Prevent the situation where errno is set by function A, then function B
is called (which is typically _(), but could be anything else) and it
overwrites errno, then errno is checked by the caller.
errno is a horrific API, and we need to be careful to save its value as
soon as a function call (which might set it) returns. i.e. Follow the
pattern:
int errsv, ret;
ret = some_call_which_might_set_errno ();
errsv = errno;
if (ret < 0)
puts (strerror (errsv));
This patch implements that pattern throughout GLib. There might be a few
places in the test code which still use errno directly. They should be
ported as necessary. It doesn’t modify all the call sites like this:
if (some_call_which_might_set_errno () && errno == ESOMETHING)
since the refactoring involved is probably more harmful than beneficial
there. It does, however, refactor other call sites regardless of whether
they were originally buggy.
https://bugzilla.gnome.org/show_bug.cgi?id=785577
Disable gio tests on Windows, fix .gitignore to not ignore
config.h.meson, and add more things to it.
Rename the library file naming and versioning to match what Autotools
outputs, e.g., libglib-2.0.so.0.5000.2 on Linux, libglib-2.0-0.dll and
glib-2.0-0.dll on Windows with MSVC.
Several more tiny fixes, more executables built and installed, install
pkg-config and m4 files, fix building of gobject tests.
Changes to gdbus-codegen to support out-of-tree builds without
environment variables set (which you can't in Meson). We now add the
build directory to the Python module search path.
These tests clear up a misunderstanding of mine: Monitoring
nonexisting files and directories *does* work with the inotify
implementation, it just has a very long timeout for scanning
for missing locations, so the test needs to take that into
account.
After the big file monitoring rewrite, we only put the IN_MOVED_FROM event
in the queue for such pairs. It matches INOTIFY_DIR_MASK and thus we call
ip_dispatch_event on it, but that function was filtering it out because
the filename in the 'from' event is the one of the temp file, not the
one we are monitoring. That name is in the 'to' event, so compare it as
well, and let the event passin that case.
There is another instance of this check in glocalfilemonitor.c, which is
corrected here as well.
https://bugzilla.gnome.org/show_bug.cgi?id=751358
The hash table stores the list of unmatched IN_MOVE_FROM events, but we
were removing entries from it when popping IN_MOVE_TO events.
Fix that up to correct a crash in nautilus due to the assertion failure
below.
https://bugzilla.gnome.org/show_bug.cgi?id=746749
Use the "interesting" value from g_file_monitor_source_handle_event() to
decide if we're currently being flooded by a stream of boring events.
The main case here is when one or more files is being written to and the
change events are all being rate-limited in the GFileMonitor frontends.
In that case, we become "bored" with the event stream and add a backoff
timeout. In the case that it is exactly one large file being written
(which is the common case) then leaving the event in the queue also lets
the kernel perform merging on it, so when we wake up, we will only see
the one event. Even in the case that the kernel is unable to perform
merging, the context switch overhead will be vastly reduced.
In testing, this cuts down on the number of wake ups during a large file
copy, by a couple orders of magnitude (ie: less than 1% of the number of
wake ups).
We generally assume that an IN_CREATE event is the start of a series of
events in which another process is doing this:
fd = creat (...) -> IN_CREATE
write (fd, ..) -> IN_MODIFY
write (fd, ..) -> IN_MODIFY
close (fd) -> IN_CLOSE_WRITE
and as such, we use the CHANGES_DONE_HINT event after CREATED in order
to show when this sequence of events has completed (ie: when we receive
IN_CLOSE_WRITE when the user closes the file).
Renaming a file into place is handled by IN_MOVED_FROM so we don't have
to worry about that.
There are many other cases, however, where a new file 'appears' in a
directory in its completed form already, and the kernel reports
IN_CREATE. Examples include mkdir, mknod, and the creation of
hardlinks. In these cases, there is no corresponding IN_CLOSE_WRITE
event and the CHANGES_DONE_HINT will have to be emitted by an arbitrary
timeout.
Try to detect some of these cases and report CHANGES_DONE_HINT
immediately.
This is not perfect. There are some cases that will not be reliably
detected. An example is if the user makes a hardlink and then
immediately deletes the original (before we can stat the new file).
Another example is if the user creates a file with O_TMPFILE. In both
of these cases, CHANGES_DONE_HINT will still eventually be delivered via
the timeout.
Remove all event merging and dispatch logic from GFileMonitor. The only
implementation of GFileMonitor outside of glib is in gvfs and it already
does these things properly.
Get rid of GLocalDirectoryMonitor. We will use a single class,
GLocalFileMonitor, for both directory and file monitoring. This will
prevent every single backend from having to create two objects
separately (eg: ginotifydirectorymonitor.c and ginotifyfilemonitor.c).
Introduce GFileMonitorSource as a thread-safe cross-context dispatch
mechanism. Put it in GLocalFileMonitor. All backends will be expected
to dispatch via the source and not touch the GFileMonitor object at all
from the worker thread.
Remove all construct properties from GLocalFileMonitor and remove the
"context" construct property from GFileMonitor. All backends must now
get the information about what file to monitor from the ->start() call
which is mandatory to implement.
Remove the implementation of rate limiting in GFileMonitor and add an
implementation in GLocalFileMonitor. gvfs never did anything with this
anyway, but if it wanted to, it would have to implement it for itself.
This was done in order to get the rate_limit field into the
GFileMonitorSource so that it could be safely accessed from the worker
thread.
Expose g_local_file_is_remote() internally for NFS detection.
With the "is_remote" functionality exposed, we can now move all
functions for creating local file monitors to a proper location in
glocalfilemonitor.c
Port the inotify backend to adjust to the changes above. None of the
other backends are ported yet. Those will come in future commits.
Remove the hardwired 1 second event queue logic from inotify-kernel and
replace it with something vastly less complicated.
Events are now reported as soon as is possible instead of after a
delay.
We still must delay IN_MOVED_FROM events in order to look for the
matching IN_MOVED_TO events, and since we want to report events in order
this means that events behind those events can also be delayed. We
limit ourselves, however:
- no more than 100 events can be delayed at a time
- no event can be delayed by more than 10ms
https://bugzilla.gnome.org/show_bug.cgi?id=627285
Our check for inotify_init1() being defined is broken. We happily
declare that inotify is supported, even if the check fails.
This was originally intended to check for inotify_init1 in the libc so
that we could fall back to inotify_init if it was not yet defined.
FreeBSD has a libinotify that emulates the inotify API via kqueue. It
installs a <sys/inotify.h> header and requires linking to -linotify. We
don't want to falsely detect working inotify in this case.
Treat the lack of inotify_init1() in the libc as a lack of inotify
support. This requires only a new libc -- we still support old kernels:
in the case that inotify1_init() fails, we fall back to inotify_init().
https://bugzilla.gnome.org/show_bug.cgi?id=724330
and start using the new start() vcall on the local monitor classes.
I only port inotify because I am uncomfortable making changes to the
other monitor backends without having a way of testing them.
https://bugzilla.gnome.org/show_bug.cgi?id=704887
GFileMonitor takes great care to sample the thread-default main context
at the time that it is created in order that events can be dispatched to
the correct thread when they come in.
The inotify GFileMonitor implementation uses a global file descriptor
shared between all watches. It has to poll this file descriptor from
somewhere so it arbitrarily picks the default main context.
The problem with that is that the user might not be running it.
Let's use the GLib worker thread for this instead. It's guaranteed to
be running if you need it, and this is exactly the sort of problem it
was meant to solve.
https://bugzilla.gnome.org/show_bug.cgi?id=704873
