The C Source, Patches and (shudder!) Bugs
mauritiusdadd
Posts: 4
Joined: 29 Jun 2015, 07:57

usb_modeswitch does not work with systemd 221

Post by mauritiusdadd » 29 Jun 2015, 08:28

Hi, this is my first post, so hello to everyone!

I'm using Archlinux on a x86_64 machine. After we recently updated to systemd 221, a lot of users reported that usb_modeswitch 2.2.1 has stopped to work (see this thread on Archlinux forums: https://bbs.archlinux.org/viewtopic.php?id=199075). I've done some tests using a Huawei E1820 ( vendor-id=12d1 and device-id=1446) and I can confirm that usb_modeswitch does not work unless you run it manually: when the device is inserted udev correctly applies the rule 40-usb_modeswitch.rules and successfully invokes /usr/lib/udev/usb_modeswitch but the latter script however will never run usb_mode_dispatcher. After playing a bit with the scripts I've found that what causes troubles is the exec command on line 76 of /usr/lib/udev/usb_modeswitch :

Code: Select all

76  exec 1<&- 2<&- 5<&- 7<&-   # <-----------------
77  (
78  count=20
79  while [ $count != 0 ]; do
80 	if [ ! -e "/usr/bin/usb_modeswitch_dispatcher" ]; then
81 		sleep 1
82 		count=$(($count - 1))
83 	else
84 		if [ -e "/etc/init/usb-modeswitch-upstart.conf" ]; then
85 			exec /sbin/initctl emit --no-wait usb-modeswitch-upstart UMS_PARAM=$1
86 		elif [ -e "/etc/systemd/system/usb_modeswitch@.service" ]; then
87 			exec /usr/bin/systemctl --no-block start usb_modeswitch@$1.service
88 		else
89 			exec /usr/bin/usb_modeswitch_dispatcher --switch-mode $1 &
90 		fi
91 		exit 0
92  	fi
93  done
94  ) &
95  exit 0
which basically (if my understanding of the exec command is correct) is needed to close the stdin, stdout and the file descriptors 5 and 7 (that however are not open here) and prevents the journal to be flooded by usb_modeswitch messages. However, It seems that now at least one of the stdin or stdout must remain opened otherwise the subsequent commands will not be executed.

I also installed the new version 2.2.2 and the result is the same: if I comment out that exec line then everything works as expected.

--edit: typo

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 29 Jun 2015, 14:56

Thanks for the report!

You have probably guessed why this strange construct was used. The subscript may have to wait during boot for a folder to become available. However, udev does not allow its processes to run for several seconds, so I tried to get the loop forked and detached from the initial script.

There may be better ways to do it, for sure. I wonder if you can test the following which should work in dash as well. Replace the whole "exec" and the enclosing brackets with this:

nohup
(
...

...
) &

This works well here (systemd v. 219), even with a "sleep 10" command inserted between the brackets.
I'm just updating my Mageia distribution, hoping that there will be a fairly new version of systemd afterwards.

mauritiusdadd
Posts: 4
Joined: 29 Jun 2015, 07:57

Re: usb_modeswitch does not work with systemd 221

Post by mauritiusdadd » 29 Jun 2015, 15:19

The nohup command works fine here too, thanks!

mauritiusdadd
Posts: 4
Joined: 29 Jun 2015, 07:57

Re: usb_modeswitch does not work with systemd 221

Post by mauritiusdadd » 29 Jun 2015, 15:45

Sorry, actually i found this error message in the log

Code: Select all

giu 29 15:41:33 530U3C systemd-udevd[17761]: starting 'usb_modeswitch '/1-1:1.0''
giu 29 15:41:33 530U3C systemd-udevd[17741]: 'usb_modeswitch '/1-1:1.0''(err) 'nohup: missing operand'
giu 29 15:41:33 530U3C systemd-udevd[17741]: 'usb_modeswitch '/1-1:1.0''(err) 'Try 'nohup --help' for more information.'
and this is how I modified the script

Code: Select all

#exec 1<&- 2<&- 5<&- 7<&-
nohup
(
IFS='/' read -r p1 p2 <<EOF
$1
EOF
PATH=/bin:/sbin:/usr/bin:/usr/sbin
count=20
while [ $count != 0 ]; do
	if [ ! -e "/usr/bin/usb_modeswitch_dispatcher" ]; then
		sleep 1
		count=$(($count - 1))
	else
		if [ -e "/etc/systemd/system/usb_modeswitch@.service" ]; then
			exec systemctl --no-block start usb_modeswitch@$p1'_'$p2.service
		elif [ -e "/etc/init/usb-modeswitch-upstart.conf" ]; then
			exec initctl emit --no-wait usb-modeswitch-upstart UMS_PARAM=$1
		else
			exec usb_modeswitch_dispatcher --switch-mode $1 &
		fi
		exit 0
	fi
done
) &
exit 0
is this correct?

--edit:

It seems that the nohup command needs an actual commad/script to work. Since the command 'exec 1<&- 2<&- 5<&- 7<&-' only influence the output redirection why not just removig it and replace the enclosing brackets with

Code: Select all

(
...

...
) > /dev/null &
?

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 29 Jun 2015, 19:18

Hmm, you're right, the nohup line has no effect.

However, I'm wary about breaking older systems ... I've read in several places that just forking subscripts to the background means they will get killed once the parent process ends, which is precisely what nohup should prevent - even if my recent tests showed that current udev versions do obviously not kill the script prematurely. Earlier versions do so definitely, that's why I started to 'abuse' the init mechanics.

Anyway, I did more tests, starting a subscript just with brackets - like you suggested - with a long wait; the script returned at once and then I immediately logged out entirely. The subscript completed successfully anyway.

I will repeat the tests in Ubuntu, but for now it looks like the subscript is not affected by the parent's lifecycle in the 'dash' version available on Mageia.

I'll post again.

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 29 Jun 2015, 23:19

None of my distributions has systemd above version 219, so I can't replay the problem. However, the findings of the Archlinux community are strongly pointing to a serious problem here.

I will issue a bugfix release of usb_modeswitch (the program package) with the change you suggested (just brackets and "&"). It worked nicely on Ubuntu too, the test script never blocked or killed its 'sibling'.

Thanks for the research! This may well spare users of other distributions plenty of headaches ...

mauritiusdadd
Posts: 4
Joined: 29 Jun 2015, 07:57

Re: usb_modeswitch does not work with systemd 221

Post by mauritiusdadd » 30 Jun 2015, 08:10

Ok, thank you very much!

dsd
Posts: 5
Joined: 04 Mar 2016, 20:04

Re: usb_modeswitch does not work with systemd 221

Post by dsd » 24 May 2016, 23:00

I think this change caused a regression.

We know that anything run by udev's RUN functionality should be fast, which is why usb-modeswitch tries to fork here. We hope that upon forking, udev will be unblocked.
But what's not made clear in the documentation is that actually udev will also block monitoring the stdout/stderr of the child process. So forking is not enough, you also need to close stdin and stderr.
See https://github.com/systemd/systemd/blob ... ent.c#L745 - note that the original process (default: case) does spawn_read (which blocks on those pipes) before calling spawn_wait (which checks if the original process went away).

This results in the following sequence of events here:
* I plug in my modem
* usb_modeswitch starts and udev is blocked at this point
* mode switch happens successfully, new device appears
* usb_modeswitch waits 1500ms to see if a driver claimed interface 0 (nothing did, since udev is still blocked)
* usb_modeswitch goes to default behaviour of loading 'option' module and forcing it to bind to the device
* usb_modeswitch exits
* udev is now unblocked, so it processes queued events for the new device. It loads the required driver huawei_cdc_ncm, but that driver claims nothing since option is driving the interfaces

So my modem is unusable since the wrong driver was loaded.

Here is a full debug log where you can see udev clearly waiting for usb_modeswitch to end before processing the new device:
https://gist.github.com/dsd/ef7afa261f6 ... d001e49c80
Line 312 is where usb_modeswitch ends and udev jumps back into life, loading the huawei_cdc_ncm module on line 564.

Now I readd "exec 1<&- 2<&-" where it used to be, new log:
https://gist.github.com/dsd/ae9a713e4e3 ... f76d1af137
Now you can see udev immediately acting on the modeswitched device, loading the huawei_cdc_ncm driver on line 554. usb_modeswitch is still running at that time, its driver bind timeout expires on line 717, and correctly does nothing given that the kernel loaded drivers.

dsd
Posts: 5
Joined: 04 Mar 2016, 20:04

Re: usb_modeswitch does not work with systemd 221

Post by dsd » 25 May 2016, 22:24

Yesterday's testing was with systemd-215.

Now I've upgraded to systemd-229 and I can see the same behaviour as what is described on arch linux above - both before and after my "fix".

Investigating further, this is because udev kills all the subprocesses of the main process started by RUN, when that main process exits. Even if they are run through the daemonize app or something equivalent. And the udev man page makes this clear now.
RUN{type}
Add a program to the list of programs to be executed after
processing all the rules for a specific event, depending on "type":

"program"
Execute an external program specified as the assigned value. If
no absolute path is given, the program is expected to live in
/lib/udev; otherwise, the absolute path must be specified.

This is the default if no type is specified.

"builtin"
As program, but use one of the built-in programs rather than an
external one.

The program name and following arguments are separated by spaces.
Single quotes can be used to specify arguments with spaces.

This can only be used for very short-running foreground tasks.
Running an event process for a long period of time may block all
further events for this or a dependent device.

Starting daemons or other long-running processes is not appropriate
for udev; the forked processes, detached or not, will be
unconditionally killed after the event handling has finished.
I think that makes it pretty clear that /lib/udev/usb_modeswitch can no longer launch usb_modeswitch_dispatcher in the background as it is currently attempting to do.

Here's my suggestion: https://gist.github.com/dsd/9f83c4830ab ... db2cf16a8f

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 26 May 2016, 22:19

Thanks for all the tedious work with that analysis !

As you may be aware, most of the problematic code has grown historically. In the beginning, I had to make sure that the dispatcher and the binary program in /usr/sbin were available, i.e. either included with the initrd or mounted and ready in the actual filesystem.

Do you think the 'wait' for availability is obsolete now? I remember faintly to have read something about /sbin and /usr/sbin being merged and always available.

I'm generally hesitant about throwing out fallback code for very old distributions, but here there may be no choice ...

dsd
Posts: 5
Joined: 04 Mar 2016, 20:04

Re: usb_modeswitch does not work with systemd 221

Post by dsd » 26 May 2016, 22:45

Josh wrote:Thanks for all the tedious work with that analysis !

As you may be aware, most of the problematic code has grown historically. In the beginning, I had to make sure that the dispatcher and the binary program in /usr/sbin were available, i.e. either included with the initrd or mounted and ready in the actual filesystem.

Do you think the 'wait' for availability is obsolete now? I remember faintly to have read something about /sbin and /usr/sbin being merged and always available.
I think its safe to assume that usb-modeswitch will never be included in the initramfs (or if it is included, it is fully included). When systemd boots it does run udev in the initramfs, but then starts a new instance of udev when switching to the real root, so even if the modem was present earlier, it should be safe to assume that the usb modeswitch rules will be processed at the right moment under the real root.

As for /usr not being available, yes that configuration is generally regarded as broken/unworkable these days, https://freedesktop.org/wiki/Software/s ... is-broken/
Fedora has indeed merged /sbin and /usr/sbin into /usr/sbin and same for lib and bin.

I don't think Debian and Ubuntu have done that yet, but default installs will have / and /usr under the same mount, so it should be safe to assume that /usr/sbin is always present. In theory on those distros you could move and configure /usr to be a separate mount, but I think you'd expect to see various problems similar to what you'd see if usb_modeswitch didn't have the retry loop in place.
I'm generally hesitant about throwing out fallback code for very old distributions, but here there may be no choice ...
Fedora adopted systemd by default in May 2011 so you're going back 5+ years before breaking something. Ubuntu had upstart in place before that.
Debian switched from sysvinit to systemd by default more recently, so that would perhaps be the most visible breakage, but if you're still using sysvinit these days then you're probably used to modern technologies not working quite right...

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 28 May 2016, 01:05

In annother context I pondered about throwing out the driver loading code altogether, or at least 'un-default' it.
This is also a leftover from history when wireless modems were a fresh thing and there was virtually no Linux support.
Now there is quite a bit of activity in that area; new modems don't take very long until they are supported from the kernel side.

I'll be away (and offline) for a few days; after my return I'll prepare a new program and data release.

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 12 Jun 2016, 14:22

I'm in the process of preparing a new data and program package release.

I have incorporated your patch, in addition to removing all driver binding code; so you should never get the wrong drivers again.

I have added a fall-back branch to the shell script though - for old systems which likely don't mind long running udev subprocesses ...

Code: Select all

PATH=/bin:/sbin:/usr/bin:/usr/sbin
init_path=`readlink /sbin/init`
if [ `basename $init_path` = "systemd" ]; then
    systemctl --no-block start usb_modeswitch@$p1'_'$p2.service
elif [ -e "/etc/init/usb-modeswitch-upstart.conf" ]; then
    initctl emit --no-wait usb-modeswitch-upstart UMS_PARAM=$1
else
    # only old distros, new udev will kill all subprocesses
    exec 1<&- 2<&- 5<&- 7<&-
    exec usb_modeswitch_dispatcher --switch-mode $1 &
fi
exit 0

borbor
Posts: 2
Joined: 23 Nov 2016, 19:38

Re: usb_modeswitch does not work with systemd 221

Post by borbor » 23 Nov 2016, 21:20

Hello,
with usb_modeswitch-2.4.0 and systemd-232 (gentoo)
similar situation as described by mauritiusdadd (first post here, Posted: Mon Jun 29, 2015 8:28 am).
Namely:
commenting out this exec line
(
# only old distros, new udev will kill all subprocesses
# exec 1<&- 2<&- 5<&- 7<&-
exec usb_modeswitch_dispatcher --switch-mode $1 &
)
removes problems;
downgrading to usb_modeswitch-2.3.0 resolves non-switching problems.



kernel: 4.8.8-gentoo
sys-apps/systemd-232 (abi_x86_64 acl gcrypt kmod lz4 pam policykit seccomp ssl)
virtual/udev-217

mmcli -m 1: ... manufacturer: 'huawei'
| model: 'E353'
| revision: '11.810.09.20.264' ....

Similar problems are reported elsewhere, eg.:
https://bugzilla.redhat.com/show_bug.cgi?id=1373274

Josh
Site Admin
Posts: 6570
Joined: 03 Nov 2007, 00:30

Re: usb_modeswitch does not work with systemd 221

Post by Josh » 23 Nov 2016, 21:46

Hmm, that would indicate that the presence of systemd was not properly detected by the script.
Otherwise that line would not have been reached.

Can you check what "/sbin/init" links to on your system?

Post Reply