External USB disk resets spontaneously.

[jhje00]

Hi.

I bought a fit-PC2 a couple of weeks ago and I am using it as a 24/7 file- and SVN-server which it is perfectly suited for. Initially I ran a fresh install of Ubuntu 8.04 server (I prefer not having a graphical interface on a server) but the hard disk performance was very poor and the system would hang occassionally with hard disk timeouts. Well, ide_generic is not a recommended driver for anything these days.

I therefore installed the fit-pc 8.04 kernel which gave fine and stable hard disk performance. Unfortunately that kernel was missing the device mapper which I need for encrypted volumes (truecrypt).

So I installed Ubuntu 9.04 server instead which runs fine out-of-the-box with good hard disk performance. It has been rock solid for a couple of days now with respect to the internal hard disk, but I have trouble with my USB external hard disk which is used for online backups.

In /var/log/messages I occasionally see this:

Code: Select all

Sep 13 17:36:57 rumleskaft kernel: [179103.970142] usb 1-5: reset high speed USB device using ehci_hcd and address 8
Sep 13 17:48:36 rumleskaft kernel: [179802.840291] usb 1-5: reset high speed USB device using ehci_hcd and address 8
Sep 13 18:06:43 rumleskaft kernel: [180889.660296] usb 1-5: reset high speed USB device using ehci_hcd and address 8
Sep 13 18:08:53 rumleskaft kernel: [181019.520293] usb 1-5: reset high speed USB device using ehci_hcd and address 8

and earlier today I even lost connection to the USB drive:

Code: Select all

Sep 13 13:30:56 rumleskaft kernel: [164642.760290] usb 1-5: reset high speed USB device using ehci_hcd and address 3
Sep 13 13:31:07 rumleskaft kernel: [164653.150291] usb 1-5: reset high speed USB device using ehci_hcd and address 3
Sep 13 13:31:17 rumleskaft kernel: [164663.430421] sd 2:0:0:0: Device offlined - not ready after error recovery
Sep 13 13:31:17 rumleskaft kernel: [164663.430459] sd 2:0:0:0: [sdb] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 13 13:31:17 rumleskaft kernel: [164663.436193] usb 1-5: USB disconnect, address 3
Sep 13 13:31:17 rumleskaft kernel: [164663.436446] sd 2:0:0:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 13 13:31:17 rumleskaft kernel: [164663.437711] lost page write due to I/O error on sdb2
Sep 13 13:31:17 rumleskaft kernel: [164663.454184] lost page write due to I/O error on sdb2
Sep 13 13:31:18 rumleskaft kernel: [164664.250123] usb 1-5: new high speed USB device using ehci_hcd and address 4
Sep 13 13:31:19 rumleskaft kernel: [164665.235006] ------------[ cut here ]------------
Sep 13 13:31:19 rumleskaft kernel: [164665.235014] WARNING: at /build/buildd/linux-2.6.28/fs/buffer.c:1186 mark_buffer_dirty+0x75/0x90()
Sep 13 13:31:19 rumleskaft kernel: [164665.235022] Modules linked in: aes_i586 aes_generic xts gf128mul dm_crypt usb_storage input_pollde$
Sep 13 13:31:19 rumleskaft kernel: [164665.235093] Pid: 27976, comm: python Not tainted 2.6.28-11-server #42-Ubuntu
Sep 13 13:31:19 rumleskaft kernel: [164665.235100] Call Trace:
Sep 13 13:31:19 rumleskaft kernel: [164665.235115]  [<c050e026>] ? printk+0x18/0x1a
Sep 13 13:31:19 rumleskaft kernel: [164665.235127]  [<c0141244>] warn_on_slowpath+0x54/0x80
Sep 13 13:31:19 rumleskaft kernel: [164665.235139]  [<c034c700>] ? vt_console_print+0x220/0x320

After unplugging the USB disk and replugging it has worked OK, but it still gives me the occasional "reset high speed USB device..." messages when I use it heavily - which it will be during backup. I have tried to increase the INQ_TIMEOUT value from 5 to 20 seconds but it has not had any effect
(doing "echo 20 > /sys/module/scsi_mod/parameters/inq_timeout").

My kernel version is

Code: Select all

  jhje@rumleskaft:~$ cat /proc/version
  Linux version 2.6.28-11-server (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #42-Ubuntu SMP Fri Apr 17 02:48:10 UTC 2009
  jhje@rumleskaft:~$

I unmounted the USB disk and put it on my old power-hungry server (a DELL PowerEdge) and stress-tested it there. It has been running for more than 3 hours doing an infinite-loop copy of GB's of files without a blip in that server's log.

That server is running a standard Ubuntu 8.04.2:

Code: Select all

jhje@viklevakle:~$ cat /proc/version
Linux version 2.6.24-23-server (buildd@crested) (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)) #1 SMP Thu Nov 27 18:45:02 UTC 2008
jhje@viklevakle:~$

Is there a known fix for this in the fit-pc kernel ? Or is it perhaps a power supply issue ? It is definitely not a thermal issue as both the disk and the fit-pc are standing right next to a fan that cools them both (I did consider possible electrical interference from the fan and moved the USB cables a comfortable distance from the fan - no effect).

Cheers
-- Jan

[jhje00]

I suspected possible power supply issues because the USB disk that I intended to use for online backup is bus powered - it draws all its power from the USB connection. This morning I therefore mounted another USB disk that has its own power supply.

After just 5 minutes of copying files I again see the USB reset every 4-5 minutes:

Code: Select all

Sep 14 08:35:17 rumleskaft kernel: [232103.562667] EXT3-fs: mounted filesystem with ordered data mode.
Sep 14 08:41:28 rumleskaft kernel: [232474.750169] usb 1-5: reset high speed USB device using ehci_hcd and address 9
Sep 14 08:46:12 rumleskaft kernel: [232758.230279] usb 1-5: reset high speed USB device using ehci_hcd and address 9
Sep 14 08:50:17 rumleskaft kernel: [233004.090269] usb 1-5: reset high speed USB device using ehci_hcd and address 9
Sep 14 08:53:10 rumleskaft kernel: [233176.940293] usb 1-5: reset high speed USB device using ehci_hcd and address 9

Can I debug this further by bumping the logging level somehow ?

Cheers
-- Jan

[Mylife]

I haven't got a clue but I googled your error message (reset high speed USB device) and found this
http://www.mail-archive.com/ubuntu-bugs ... 68416.html

Devices for the kit to the disc-Rules are disabled.

The command is a console with administrator privileges:
mv /lib/udev/rules.d/95-devkit-disks.rules{,.disabled} -> [ENTER]

"Linux version 2.6.28-11-server"
==> You might also upgrade your kernel with the last recommended one (sudo apt-get upgrade)

[jhje00]

Hi Mylife.

Thanks for the pointer. That was a thing I hadn't tried before. However, I don't seem to have devkit or whatever it's called running on my machine. However, I tried stopping udev before I did the stress-test by running "/etc/init.d/udev stop" but it had no effect - still see the resets.

I then tried to decrease the max. sector parameter by doing

echo 128 > /sys/block/sdc/device/max_sectors (it was 240)

and even tried

echo 64 > /sys/block/sdc/device/max_sectors

but no change. Still seeing resets. I will try to upgrade the kernel from 2.6.28.11-server to 2.6.28.15-server and see if that makes a difference.

[jhje00]

Upgraded my kernel to 2.6.28-15. Still seeing resets .

This means that I cannot use USB disks. Cannot make backups. Disaster pending - Highly annoying.

Can anyone reproduce this ? To stress-test I copy my home directory (some GBs) repeatedly from the internal hard disk to the external USB disk:

while true; do cp -r /home/jhje/* /mnt/usb1/test/; done

and in parallel I monitor /var/log/messages. After 5-10 minutes I start seeing USB resets.

Kind regards
-- Jan

[Mylife]

Last try : did you choose GV3 instead of C-States in the BIOS ?
C-States hangs network on Linux, maybe it hangs USB too

[jhje00]

Bingo! Enter BIOS setup, set Processor Speed to "GV3 only" and reboot. I have now been running the external USB disk at full throttle for two hours without a blip in the log and it is still chucking along.

Thank you so much for this essential tip !

Cheers
-- Jan

[Mylife]

I'm grateful to prj too : http://www.fit-pc2.com/forum/viewtopic.php?f=9&t=361