Accepting request 121292 from Base:System

New nfs-utils for Factory - and hopefully 12.2.

- nfs upstream version 1.2.5.  Lots of bug fixes
  and improved support for pNFS.
  Adds nfsdcld daemon on osd_login script.
  Removed some very out-of-date documentation.

OBS-URL: https://build.opensuse.org/request/show/121292
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/nfs-utils?expand=0&rev=94

README.NFSv4

@@ -1,5 +1,5 @@
 NFSv4 README
-Last updated: 15 June 2006
+Last updated: 17 May 2012
 
 0. Contents:
 -----------
@@ -39,7 +39,7 @@ Last updated: 15 June 2006
 1. Overview:
 ------------
 
-The Network File System Version 4 (NFSv4) is a new distributed file system 
+The Network File System Version 4 (NFSv4) is a distributed file system 
 similar to previous versions of NFS in its straightforward design, and 
 independence of transport protocols and operating systems for file access in a 
 heterogeneous network. Unlike earlier versions of NFS, the new protocol 
@@ -52,7 +52,8 @@ Note: NFSv4 ACLs and krb5p (Kerberos Privacy) are currently not supported
 1.1 The Purpose of this document
 ________________________________
 
-This document is intended as a step-by-step guide to setup NFSv4 on SLES 10. 
+This document is intended as a step-by-step guide to setup NFSv4 on
+openSUSE 12.
 It discusses NFSv4 server and client configuration.
 
 
@@ -61,14 +62,11 @@ It discusses NFSv4 server and client configuration.
 
 For NFSv4 server:
  
-1)  Edit /etc/exports to have an entry similar to the one below: 
+1)  /etc/exports does not require any special entries to work with
-
+    NFSv4.  Earlier SUSE releases required 'fsid=0' on precisely one
-    /export  <client_ip/hostname/wildcard>(rw,fsid=0,sync,no_root_squash)
+    entry, and 'bind=' annotations on others.  This is no longer required
-
+    and should be removed.  It is still supported, so there is no need
-    (i)  fsid=0 is a must. 
+    to change /etc/exports when upgrading to openSUSE 12.
-    (ii) Replace "/export" with file tree that needs to be nfs-exported and
-    the <client_ip/hostname/wildcard> with client's ip or hostname or *. 
-    (* means any client)
 
 2)  Edit /etc/idmapd.conf to modify the default "Domain" to contain your
     DNS domain name.
@@ -124,88 +122,11 @@ ___________________________
 
 There are three main configuration files you will need to edit to set up an 
 NFSv4 server: 
-/etc/exports, /etc/sysconfig/nfs and /etc/idmapd.conf.
+/etc/sysconfig/nfs and /etc/idmapd.conf.
-we will describe the first two here as idmapd.conf is done in previous section.
+we will describe the first here as idmapd.conf is done in previous section.
 
 
-4.1.1 /etc/exports
+4.1.1 /etc/sysconfig/nfs
-==================
-
-This file contains a list of entries; each entry indicates a volume that is
-shared and how it is shared. The /etc/exports file format is slightly 
-different from previous versions. A sample exports entry looks like this.
-
-/export  *(rw,fsid=0,no_subtree_check,sync,no_root_squash)
-
-Note that:
-
-i)  fsid - The value 0 has a special meaning when use with NFSv4. NFSv4 has a 
-    concept of a root of the overall exported filesystem. The export point 
-    exported with fsid=0 will be used as this root.
-    There must be at least one entry with fsid=0. (this will be pseudo file
-    system's /)
-
-ii) The method used to mount multiple exported trees is different. NFSv4 uses 
-    the concept of pseudo filesystem to give a single file system view to the 
-    client with a pseudo-"/" as root of the filesystem tree. To illustrate,
-
-    Suppose we have
- 
-    /path1/volume1
-    /path2/volume2
-
-    as two filesystem trees on the server that need to be exported, then 
-    Firstly, these need to be bound to another name under /export directory 
-    using mount command's bind option. This is done as : 
-	mount --bind <old dir> /export/<new dir> 
-    i.e. in our example:
-
-    	#mount --bind /path1/volume1 /export/volume1
-    	#mount --bind /path2/volume2 /export/volume2
-
-    will bind these local filesystem trees to their local new names. 
-    Then these two exported filesystems (with their newly bound paths) are 
-    entered into /etc/exports with their respective independent options.
-    i.e. /etc/exports would contain -
-
-    /export/volume1 *(<options_to_be_filled>)
-    /export/volume2 *(<options_to_be_filled>)
-    
-iii)If both a directory and its subdirectory residing on different file systems
-    need to be exported, then the option 'nohide' must be appropriately used.
-    /export and  /export/subdir are on differnt file systems
-    and both need to be exported to same client then
-
-    /export         <client>(<options>)
-    /export/subdir  <client>(<options>,nohide)
- 
-    must be done so that the client can see the contents of subdir too.
-    Though this is not specific to NFSv4, it is seen as a common use case
-    scenario and is included here.
-    'man exports' has more info.
-   
-iv) Currently Yast2's nfs-server module can only be used as a subsitute
-    for manually editing the /etc/exports. Fully functional yast with other
-    configuration editing (idmapd etc) is work in progress.
-
-v)  In case of different kind of exports for the same exported path the
-    syntax that must be followed is either of the following
-    /export  host1(<options>) host2(<options>)
-    (or)
-    /export  host1(<options>)
-    /export  host2(<options>)
-
-
-
-4.1.2 Co-existing NFSv3 and NFSv4 exports for same file systems
-===============================================================
-
-NFSv4 current linux implementation caters to serving NFSv2 and NFSv3 clients
-too. The /etc/exports can contain both type of export entries even for the 
-same filesystem trees being exported.
-
-
-4.1.3 /etc/sysconfig/nfs
 =========================
 
 /etc/sysconfig/nfs is another NFS server configuration file. Here the number
@@ -358,13 +279,12 @@ Sample configuration
 
 Typical entries for kerberos security mode looks like these:
 
-/export gss/krb5(rw,fsid=0,insecure,no_subtree_check,sync,no_root_squash)
+/export gss/krb5(rw,insecure,no_subtree_check,sync,no_root_squash)
-/export gss/krb5i(rw,fsid=0,insecure,no_subtree_check,sync,no_root_squash)
+/export gss/krb5i(rw,insecure,no_subtree_check,sync,no_root_squash)
 
 Note: 
-i)   krb5p (Privacy) is currently not supported.
 
-ii)  option 'insecure' - The insecure option in this entry also allows clients 
+i)   option 'insecure' - The insecure option in this entry also allows clients 
      with NFS implementations that don't use a reserved port for NFS. So it is 
      advisable *NOT* to use this option unless you have a kerberised set up or 
      you know what you are doing.
@@ -446,14 +366,10 @@ _______________________________________________
 5. showmount -e <server name>
    to check mount information on NFS server
 
-6. Make sure that one and only one path is exported 
+6. If users are not mapped properly check whether idmapd is running in both 
-   with fsid=0. 
-   Refer Pseudofilesystems (point (iii) in Section 3.2.1) for more information.
-  
-7. If users are not mapped properly check whether idmapd is running in both 
    server & client and dns domain name is properly configured.
 
-8. If you unable to mount, check for the correctness of the exports file entry.
+7. If you unable to mount, check for the correctness of the exports file entry.
 
 
 6.2 Check list to ensure kerberos is working properly 

mkdir-sbin

@@ -0,0 +1,34 @@
+commit 03bb227402ab023f4badb515022d49f82e01ff8d
+Author: Neil Brown <neilb@suse.de>
+Date:   Thu May 17 16:40:40 2012 +1000
+
+    osd_login - ensure /sbin is created before installation.
+    
+    If we use a more standard approach to describing the osd_login
+    script, the automake infrastructure will create /sbin before
+    attempting installation.
+    This is important for: make DESTDIR=/empty-dir install
+    
+    Signed-off-by: NeilBrown <neilb@suse.de>
+
+diff --git a/utils/osd_login/Makefile.am b/utils/osd_login/Makefile.am
+index adc493a..d17ffa7 100644
+--- a/utils/osd_login/Makefile.am
++++ b/utils/osd_login/Makefile.am
+@@ -1,12 +1,9 @@
+ ## Process this file with automake to produce Makefile.in
+ 
+-OSD_LOGIN_FILES= osd_login
++# These binaries go in /sbin (not /usr/sbin), and that cannot be
++# overriden at config time.
++sbindir = /sbin
+ 
+-EXTRA_DIST= $(OSD_LOGIN_FILES)
+-
+-all-local: $(OSD_LOGIN_FILES)
+-
+-install-data-hook:
+-	$(INSTALL) --mode 755 osd_login $(DESTDIR)/sbin/osd_login
++sbin_SCRIPTS = osd_login
+ 
+ MAINTAINERCLEANFILES = Makefile.in

nfs-utils-1.2.5.tar.bz2

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3595ed4f6ee5c13514540861ad5e89907aa4fd6897bfa99577515233e4745a9b
-size 698148

nfs-utils-1.2.6.tar.bz2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70267288500408495085b7c54a667c4e8cfde5882f3193da095fc7dad2da0b83
+size 755305

nfs-utils-page_size.patch

@@ -1,11 +0,0 @@
---- utils/blkmapd/device-process.c.orig
-+++ utils/blkmapd/device-process.c
-@@ -296,7 +296,7 @@ decode_blk_volume(uint32_t **pp, uint32_
- 		off_t stripe_unit = vol->param.bv_stripe_unit;
- 		/* Check limitations imposed by device-mapper */
- 		if ((stripe_unit & (stripe_unit - 1)) != 0
--		    || stripe_unit < (off_t) (PAGE_SIZE >> 9))
-+		    || stripe_unit < (off_t) (sysconf(_SC_PAGESIZE) >> 9))
- 			return -EIO;
- 		BLK_READBUF(p, end, 4);
- 		READ32(vol->bv_vol_n);

nfs-utils.changes

@@ -1,3 +1,11 @@
+-------------------------------------------------------------------
+Thu May 17 07:04:58 UTC 2012 - nfbrown@suse.com
+
+- nfs upstream version 1.2.5.  Lots of bug fixes
+  and improved support for pNFS.
+  Adds nfsdcld daemon on osd_login script.
+  Removed some very out-of-date documentation.
+
 -------------------------------------------------------------------
 Wed May  2 01:42:23 UTC 2012 - nfbrown@suse.com
 

nfs-utils.spec

@@ -21,6 +21,7 @@ BuildRequires:  device-mapper-devel
 BuildRequires:  e2fsprogs-devel
 BuildRequires:  gcc-c++
 BuildRequires:  krb5-devel
+BuildRequires:  libevent-devel
 BuildRequires:  libgssglue-devel >= 0.3
 BuildRequires:  libmount-devel
 BuildRequires:  librpcsecgss
@@ -28,17 +29,13 @@ BuildRequires:  libtirpc-devel
 BuildRequires:  libtool
 BuildRequires:  nfsidmap-devel >= 0.24
 BuildRequires:  pkgconfig
+BuildRequires:  sqlite3-devel
 BuildRequires:  tcpd-devel
-%if 0%{?suse_version} > 1100
-BuildRequires:  libevent-devel
-%else
-BuildRequires:  libevent
-%endif
 Url:            http://nfs.sourceforge.net
 Summary:        Support Utilities for Kernel nfsd
 License:        GPL-2.0+
 Group:          Productivity/Networking/NFS
-Version:        1.2.5
+Version:        1.2.6
 Release:        0
 BuildRoot:      %{_tmppath}/%{name}-%{version}-build
 PreReq:         %fillup_prereq %insserv_prereq
@@ -60,8 +57,8 @@ Source11:       idmapd.conf
 Source12:       start-statd
 Source13:       nfs-utils.rpmlintrc
 Patch0:         nfs-utils-1.0.7-bind-syntax.patch
-Patch1:         warn-nfs-udp.patch
+Patch1:         remove_pretty_sig.patch
-Patch2:         nfs-utils-page_size.patch
+Patch2:         mkdir-sbin
 Suggests:       python-base
 
 %description
@@ -139,11 +136,10 @@ Authors:
 %setup -q -n nfs-utils-%{version} -a 1
 %patch0 -p1
 %patch1 -p1
-%patch2
+%patch2 -p1
 cp %{S:6} .
 
 %build
-%{?suse_update_config:%{suse_update_config -f }}
 rm -f configure; autoreconf -fi
 CFLAGS="$RPM_OPT_FLAGS -fPIE -fno-strict-aliasing" LDFLAGS="-pie" ./configure \
 	--mandir=%{_mandir} \
@@ -151,6 +147,7 @@ CFLAGS="$RPM_OPT_FLAGS -fPIE -fno-strict-aliasing" LDFLAGS="-pie" ./configure \
 	--enable-nfsv4 \
 	--enable-gss \
 	--enable-ipv6 \
+	--enable-nfsdcld \
 	--enable-mount \
 	--enable-libmount-mount \
 	--enable-mountconfig \
@@ -266,6 +263,7 @@ fi
 /sbin/mount.nfs4
 /sbin/umount.nfs
 /sbin/umount.nfs4
+/sbin/osd_login
 /usr/sbin/gss_clnt_send_err
 /usr/sbin/gss_destroy_creds
 %attr(0755,root,root) /usr/sbin/mountstats
@@ -318,6 +316,7 @@ fi
 /usr/sbin/rpc.mountd
 /usr/sbin/rpc.nfsd
 /usr/sbin/rpc.svcgssd
+/usr/sbin/nfsdcld
 %{_mandir}/man5/exports.5.gz
 %{_mandir}/man7/nfsd.7.gz
 %{_mandir}/man8/exportfs.8.gz
@@ -326,6 +325,7 @@ fi
 %{_mandir}/man8/rpc.mountd.8.gz
 %{_mandir}/man8/rpc.nfsd.8.gz
 %{_mandir}/man8/rpc.svcgssd.8.gz
+%{_mandir}/man8/nfsdcld.8.gz
 /usr/share/omc/svcinfo.d/nfs-kernel-server.xml
 %config(noreplace) /var/lib/nfs/xtab
 %config(noreplace) /var/lib/nfs/etab
@@ -334,6 +334,6 @@ fi
 
 %files -n nfs-doc
 %defattr(-,root,root)
-%doc nfs/*.html nfs/*.ps linux-nfs/* README.NFSv4
+%doc nfs/*.html nfs/*.ps README.NFSv4
 
 %changelog

remove_pretty_sig.patch

@@ -0,0 +1,79 @@
+From: Jim Rees <rees@umich.edu>
+Date: Tue, 27 Mar 2012 10:18:39 -0400
+Subject: [PATCH 1/2] remove pretty_sig()
+
+This really only works against EMC servers.  There is at least one server
+that returns unprintable signatures, which fill the log with garbage (the
+spec does not mandate printable signatures).  It could be made more generic,
+for example by checking each byte for isprint().  But the signatures are
+really only of interest to developers, not admins, so it seems better to
+just remove it.
+
+Signed-off-by: Jim Rees <rees@umich.edu>
+Acked-by: NeilBrown <neilb@suse.de>
+---
+ utils/blkmapd/device-process.c |   29 -----------------------------
+ 1 file changed, 29 deletions(-)
+
+diff --git a/utils/blkmapd/device-process.c b/utils/blkmapd/device-process.c
+index 652a7a8..5fe3dff 100644
+--- a/utils/blkmapd/device-process.c
++++ b/utils/blkmapd/device-process.c
+@@ -49,28 +49,6 @@
+ 
+ #include "device-discovery.h"
+ 
+-static char *pretty_sig(char *sig, uint32_t siglen)
+-{
+-	static char rs[100];
+-	uint64_t sigval;
+-	unsigned int i;
+-
+-	if (siglen <= sizeof(sigval)) {
+-		sigval = 0;
+-		for (i = 0; i < siglen; i++)
+-			sigval |= ((unsigned char *)sig)[i] << (i * 8);
+-		sprintf(rs, "0x%0llx", (unsigned long long) sigval);
+-	} else {
+-		if (siglen > sizeof rs - 4) {
+-			siglen = sizeof rs - 4;
+-			sprintf(&rs[siglen], "...");
+-		} else
+-			rs[siglen] = '\0';
+-		memcpy(rs, sig, siglen);
+-	}
+-	return rs;
+-}
+-
+ uint32_t *blk_overflow(uint32_t * p, uint32_t * end, size_t nbytes)
+ {
+ 	uint32_t *q = p + ((nbytes + 3) >> 2);
+@@ -109,9 +87,6 @@ static int decode_blk_signature(uint32_t **pp, uint32_t * end,
+ 		 * for mapping, then thrown away.
+ 		 */
+ 		comp->bs_string = (char *)p;
+-		BL_LOG_INFO("%s: si_comps[%d]: bs_length %d, bs_string %s\n",
+-			    __func__, i, siglen,
+-			    pretty_sig(comp->bs_string, siglen));
+ 		p += ((siglen + 3) >> 2);
+ 	}
+ 	*pp = p;
+@@ -152,10 +127,6 @@ read_cmp_blk_sig(struct bl_disk *disk, int fd, struct bl_sig_comp *comp)
+ 	}
+ 
+ 	ret = memcmp(sig, comp->bs_string, siglen);
+-	if (!ret)
+-		BL_LOG_INFO("%s: %s sig %s at %lld\n", __func__, dev_name,
+-			    pretty_sig(sig, siglen),
+-			    (long long)comp->bs_offset);
+ 
+  out:
+ 	if (sig)
+-- 
+1.7.9.5
+
+--
+To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at  http://vger.kernel.org/majordomo-info.html
+

warn-nfs-udp.patch

@@ -1,160 +0,0 @@
----
- utils/mount/nfs.man    |   83 +++++++++++++++++++++++++++++++++++++++++++++++++
- utils/mount/nfsmount.c |    3 +
- utils/mount/stropts.c  |   13 +++++++
- 3 files changed, 99 insertions(+)
-
---- nfs-utils-1.2.4.orig/utils/mount/nfs.man
-+++ nfs-utils-1.2.4/utils/mount/nfs.man
-@@ -503,6 +503,8 @@ Specifying a netid that uses TCP forces
- command and the NFS client to use TCP.
- Specifying a netid that uses UDP forces all traffic types to use UDP.
- .IP
-+.B Before using NFS over UDP, please refer to the section WARNINGS below.
-+.IP
- If the
- .B proto
- mount option is not specified, the
-@@ -517,6 +519,8 @@ The
- option is an alternative to specifying
- .BR proto=udp.
- It is included for compatibility with other operating systems.
-+.IP
-+.B Before using NFS over UDP, please refer to the section WARNINGS below.
- .TP 1.5i
- .B tcp
- The
-@@ -975,6 +979,8 @@ in  a single  frame) is advised.  This r
- the loss of a single MTU-sized network frame results in the loss of
- an entire large read or write request.
- .P
-+Please see also the WARNINGS section below.
-+.P
- TCP is the default transport protocol used for all modern NFS
- implementations.  It performs well in almost every conceivable
- network environment and provides excellent guarantees against data
-@@ -1566,6 +1572,83 @@ export pathname, but not both, during a
- merges the mount option
- .B ro
- with the mount options already saved on disk for the NFS server mounted at /mnt.
-+.SH WARNINGS
-+Using NFS over UDP on high-speed links such as Gigabit
-+.BR "can cause silent data corruption" .
-+.P
-+The problem can be triggered at high loads, and is caused by problems in
-+IP fragment reassembly. NFS read and writes typically transmit UDP packets
-+of 4 Kilobytes or more, which have to be broken up into several fragments
-+in order to be sent over the Ethernet link, which limits packets to 1500
-+bytes by default. This process happens at the IP network layer and is
-+called fragmentation.
-+.P
-+In order to identify fragments that belong together, IP assigns a 16bit
-+.I IP ID
-+value to each packet; fragments generated from the same UDP packet
-+will have the same IP ID. The receiving system will collect these
-+fragments and combine them to form the original UDP packet. This process
-+is called reassembly. The default timeout for packet reassembly is
-+30 seconds; if the network stack does not receive all fragments of
-+a given packet within this interval, it assumes the missing fragment(s)
-+got lost and discards those it already received.
-+.P
-+The problem this creates over high-speed links is that it is possible
-+to send more than 65536 packets within 30 seconds. In fact, with
-+heavy NFS traffic one can observe that the IP IDs repeat after about
-+5 seconds.
-+.P
-+This has serious effects on reassembly: if one fragment gets lost,
-+another fragment
-+.I from a different packet
-+but with the
-+.I same IP ID
-+will arrive within the 30 second timeout, and the network stack will
-+combine these fragments to form a new packet. Most of the time, network
-+layers above IP will detect this mismatched reassembly - in the case
-+of UDP, the UDP checksum, which is a 16 bit checksum over the entire
-+packet payload, will usually not match, and UDP will discard the
-+bad packet.
-+.P
-+However, the UDP checksum is 16 bit only, so there is a chance of 1 in
-+65536 that it will match even if the packet payload is completely
-+random (which very often isn't the case). If that is the case,
-+silent data corruption will occur.
-+.P
-+This potential should be taken seriously, at least on Gigabit
-+Ethernet.
-+Network speeds of 100Mbit/s should be considered less
-+problematic, because with most traffic patterns IP ID wrap around
-+will take much longer than 30 seconds.
-+.P
-+It is therefore strongly recommended to use
-+.BR "NFS over TCP where possible" ,
-+since TCP does not perform fragmentation.
-+.P
-+If you absolutely have to use NFS over UDP over Gigabit Ethernet,
-+some steps can be taken to mitigate the problem and reduce the
-+probability of corruption:
-+.TP +1.5i
-+.I Jumbo frames:
-+Many Gigabit network cards are capable of transmitting
-+frames bigger than the 1500 byte limit of traditional Ethernet, typically
-+9000 bytes. Using jumbo frames of 9000 bytes will allow you to run NFS over
-+UDP at a page size of 8K without fragmentation. Of course, this is
-+only feasible if all involved stations support jumbo frames.
-+.IP
-+To enable a machine to send jumbo frames on cards that support it,
-+it is sufficient to configure the interface for a MTU value of 9000.
-+.TP +1.5i
-+.I Lower reassembly timeout:
-+By lowering this timeout below the time it takes the IP ID counter
-+to wrap around, incorrect reassembly of fragments can be prevented
-+as well. To do so, simply write the new timeout value (in seconds)
-+to the file
-+.BR /proc/sys/net/ipv4/ipfrag_time .
-+.IP
-+A value of 2 seconds will greatly reduce the probability of IPID clashes on
-+a single Gigabit link, while still allowing for a reasonable timeout
-+when receiving fragmented traffic from distant peers.
- .SH FILES
- .TP 1.5i
- .I /etc/fstab
---- nfs-utils-1.2.4.orig/utils/mount/nfsmount.c
-+++ nfs-utils-1.2.4/utils/mount/nfsmount.c
-@@ -264,6 +264,9 @@ parse_options(char *old_opts, struct nfs
- 				if (!strcmp(opteq+1, "udp")) {
- 					nfs_pmap->pm_prot = IPPROTO_UDP;
- 					mnt_pmap->pm_prot = IPPROTO_UDP;
-+					fprintf(stderr,
-+						"Using NFS over UDP can cause data corruption.\n"
-+						"Please refer to the WARNINGS section of the nfs(5) manual page.\n");
- #if NFS_MOUNT_VERSION >= 2
- 					data->flags &= ~NFS_MOUNT_TCP;
- 				} else if (!strcmp(opteq+1, "tcp") &&
---- nfs-utils-1.2.4.orig/utils/mount/stropts.c
-+++ nfs-utils-1.2.4/utils/mount/stropts.c
-@@ -567,6 +567,8 @@ static int nfs_sys_mount(struct nfsmount
- {
- 	char *options = NULL;
- 	int result;
-+	char *proto;
-+	static int once = 0;
- 
- 	if (mi->fake)
- 		return 1;
-@@ -575,6 +577,17 @@ static int nfs_sys_mount(struct nfsmount
- 		errno = EIO;
- 		return 0;
- 	}
-+	if (po_contains(mi->options, "udp"))
-+		proto = "udp";
-+	else
-+		proto = po_get(mi->options, "proto");
-+	if (proto && strcmp(proto, "udp") == 0 && !once) {
-+		fprintf(stderr,
-+			"Using NFS over UDP can cause data corruption.\n"
-+			"Please refer to the WARNINGS section of the nfs(5) manual page.\n");
-+		once=1;
-+	}
-+
- 
- 	result = mount(mi->spec, mi->node, mi->type,
- 			mi->flags & ~(MS_USER|MS_USERS), options);