The woes of 520-byte sectors

A couple years ago, I bought a 12G/s SAS disk to see if I could get it to work with the RAID controller with external SFF-8088 ports which came with the system I got while I was working at The Linux Foundation. I got an enclosure to go with it because I was enthusiastic and optimistic about my ability to get things all set up. My plan was to take the 1T disks I had in the storage server I had been using before, but which had since failed and to put them into this new enclosure. I got a couple of SFF-8088 cables, and the enclosure had some SFF-8088 to 7-pin SATA break-outs come with it. I bought an additional dual SFF-8088 to SFF-8087 adapter and a couple of SFF-8087 to 4x SFF-8482 cables. I got the 8482s because I imagined that these would be required to take full advantage of the 6TB 12G/s SAS disk mentioned above.

Unfortunately, when I attached the disks to the controller, none of them worked and I gave up frustrated and disillusioned for a good long time. Last week was spring break, and I took a few days off from work to hang out with my girls. While we weren’t all goofing off together, I was puttering around with a new HBA I got to replace the apparently non-functional HP RAID controller which came with the server.

However, when I removed the HP RAID controller and put in the LSI 9201-16E HBA I got to replace it with, there was again no SCSI love. The lsscsi command showed nothing. But the lspci command showed the controller, and I started digging through documentation to figure out what might be wrong. I eventually stumbled upon a firmware flashing ISO disk image on the Broadcom site. Broadcom apparently purchased LSI not too long ago and is now responsible for managing documentation and downloads for the legacy devices, of which I am now a proud owner.

I was able to build a USB disk from this ISO using unetbootin and emacs. I had to modify the syslinux.cfg file to correct the path and case of the filenames. And for some reason, unetbootin replaced the filename of the disk image from which freedos was supposed to boot with something useless. In any case, after corrections, the entry in syslinux.cfg looks like this:

label ubnentry20
menu label 9201_16e
kernel /BOOT/ISOLINUX/MEMDISK
append initrd=/LSI/IMG/9201_16E.IMG

Unfortunately, the firmware on the 9201_16E.IMG fat filesystem was old, so I loop mounted it and wrote a more recent version of the firmware, mptsas2.rom and sas2flsh.exe to it. I’ve uploaded it to 9201_16E.IMG in case anyone might find it useful. While I’m at it, I suppose I should put the whole usb boot image up for those who might need it. See lsi-flash-usb.img.bz2. To use this disk image, decompress the .bz2 file and dd it to your USB block device, for instance /dev/sdb in my case:

$ bunzip2 /tmp/lsi-flash-usb.img.bz2 | sudo dd of=/dev/sdb

My Proliant DL370 G6 server doesn’t have a fancy new UEFI system, so I used this boot disk to get access to the SAS controller in real mode. Once I booted it and selected 9201_16e (the 21st option from the top), I had to rapidly press the up or down arrow and select the EMM386 option to load the driver. For those playing along at home, I placed the more recent LSI controller firmware and BIOS in the \CJ directory, which has the same structure as the \LSI directory from the original disk image. You should be able to

cd \CJ

and run

9201.bat

Answer the prompts, and afterwards you should have a controller with P20 firmware and BIOS written to it.

Daunting as this was, it only got me most of the way through the process. All of my SATA disks were working at this point, but the SAS disk was not. After some sleuthing, I discovered that the disk was SCSI formatted to 520 byte sectors. Linux does not accept sector sizes that are not a power of 2, and so i was unable to do anything with the disk using fdisk, kpartx, etc. Research pointed me to the sg_format utility, but I was unable to make that work. Here’s the output of the command:

$ sudo sg_format --format --size 512 -vv /dev/sg4
open /dev/sg4 with flags=0x802
    inquiry cdb: 12 00 00 00 24 00 
    SEAGATE   DKS2F-H6R0SS      7FA6   peripheral_type: disk [0x0]
      PROTECT=0
    inquiry cdb: 12 01 00 00 24 00 
    inquiry cdb: 12 01 80 01 00 00 
      Unit serial number: Z4D0M2BF0000W515S4WH
    inquiry cdb: 12 01 83 01 00 00 
      LU name: 5000c50062ba7973
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 28 bytes
Mode Sense (block descriptor) data, prior to changes:
Mode sense number of blocks maxed out, set longlba
    mode sense (10) cdb: 5a 10 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 36 bytes
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=11473076960 [0x2abd942e0]
  Block size=520 [0x208]
    mode select (10) cdb: 55 11 00 00 00 00 00 00 22 00 
    mode select (10) parameter list
00 00 00 00 01 00 00 10  00 00 00 00 00 00 00 00
00 00 00 00 00 00 04 00  01 0a 0c 14 ff 00 00 00
05 00
mode select (10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
  Descriptor type: Sense key specific: Field pointer:
        Error in Command: byte 7 bit 7
  Descriptor type: Field replaceable unit code: 0x5
  Descriptor type: Vendor specific [0x80]
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 
 Raw sense data (in hex):
        72 05 1a 00 00 00 00 1c  02 06 00 00 cf 00 07 00
        03 02 00 05 80 0e 00 00  00 00 00 00 00 00 00 00
        00 00 00 00
MODE SELECT command: Illegal request sense key, apart from Invalid opcode

I was about to give up, but for some reason did not. I tried to find firmware and read any documentation about my disk, a Seagate ST6000NM0024. Eventually I found a github repo called ToolBin. In this repo there is a program named SeaChest_Format_121_1183_64 which seems to use a proprietary SCSI format command to tell the drive to format itself using 512-byte sectors. I can’t say for certain yet that it works, but it did not immediately fail as sg_format did. The command I used (and this will vary for you, depending on what sg_scan -i tells you is your scsi generic device) is:

./SeaChest_Format_121_1183_64 -d /dev/sg5 --fastFormat 1 --formatUnit 512 --confirm I-understand-this-command-will-erase-all-data-on-the-drive

Assuming this command results in success after the 12 hour run, I will purchase another 7 of these drives and fill out my disk array!

Sun Apr 14 13:39:48 PDT 2019
        Format Unit Progress = 3.61%

Thanks to the #linux-raid channel for helping me through this!

Update:

Mon Apr 15 08:16:00 PDT 2019
        Format Unit Progress = 99.97%
Mon Apr 15 08:17:00 PDT 2019
        Format Unit Progress = 100.00%

[70874.677389] sd 3:0:2:0: device_block, handle(0x0012)
[70876.176199] sd 3:0:2:0: device_unblock and setting to running, handle(0x0012)
[70876.249541] mpt2sas_cm0: removing handle(0x0012), sas_addr(0x5000c50062ba7971)
[70876.253344] mpt2sas_cm0: removing : enclosure logical id(0x500062b20059ecc0), slot(1)
[70886.680412] scsi 3:0:3:0: Direct-Access     SEAGATE  DKS2F-H6R0SS     7FA6 PQ: 0 ANSI: 5
[70886.684166] scsi 3:0:3:0: SSP: handle(0x0012), sas_addr(0x5000c50062ba7971), phy(2), device_name(0x5000c50062ba7970)
[70886.688833] scsi 3:0:3:0: SSP: enclosure_logical_id(0x500062b20059ecc0), slot(1)
[70886.750049] sd 3:0:3:0: Attached scsi generic sg5 type 0
[70886.750143] sd 3:0:3:0: [sdd] Spinning up disk...
[70887.765498] .
[70888.789522] .
[70889.813591] .
[70890.837718] .
[70891.861715] .
[70892.885876] .
[70893.909889] .
[70894.933912] .
[70895.957989] .
[70896.982028] .
[70898.006133] .
[70899.030215] .
[70900.054269] .
[70901.078340] .
[70902.102359] .
[70903.478491] .
[70904.502557] .
[70906.823608] ready
[70906.826165] sd 3:0:3:0: [sdd] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[70906.829779] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[70906.834708] sd 3:0:3:0: [sdd] Write Protect is off
[70906.837364] sd 3:0:3:0: [sdd] Mode Sense: 13 00 10 08
[70906.838142] sd 3:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[70906.867106] sd 3:0:3:0: [sdd] Attached SCSI disk

$ sudo ./SeaChest_Info_141_1183_64  -d /dev/sg5 -i
==========================================================================================
 SeaChest_Info - Seagate drive utilities - NVMe Enabled
 Copyright (c) 2014-2018 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 SeaChest_Info Version: 1.4.1-1_18_3 X86_64
 Build Date: Oct 18 2018
 Today: Mon Apr 15 14:52:33 2019
==========================================================================================

/dev/sg5 - DKS2F-H6R0SS - Z4D0M2BF0000W515S4WH - SCSI
        Vendor ID: SEAGATE 
        Model Number: DKS2F-H6R0SS    
        Serial Number: Z4D0M2BF0000W515S4WH
        Firmware Revision: 7FA6
        World Wide Name: 5000C50062BA7973
        Copyright: Copyright (c) 2014 Seagate All rights reserved 
        Drive Capacity (TB/TiB): 6.00/5.46
        Temperature Data:
                Current Temperature (C): 45
                Highest Temperature (C): Not Reported
                Lowest Temperature (C): Not Reported
        Power On Time:  21 days 11 hours 8 minutes 
        Power On Hours: 515.13
        MaxLBA: 11721045167
        Native MaxLBA: Not Reported
        Logical Sector Size (B): 512
        Physical Sector Size (B): 4096
        Sector Alignment: 0
        Rotation Rate (RPM): 7200
        Form Factor (inch): 3.5
        Last DST information:
                DST has never been run
        Long Drive Self Test Time:  10 hours 2 minutes 
        Interface speed:
                Port 0 (Current Port)
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): 6.0
                Port 1
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): Not Reported
        Annualized Workload Rate (TB/yr): 0.00
        Total Bytes Read (MB): 6.39
        Total Bytes Written (MB): 14.99
        Encryption Support: Not Supported
        Cache Size (MiB): Not Reported
        Read Look-Ahead: Not Supported
        Write Cache: Disabled
        SMART Status: Good
        ATA Security Information: Not Supported
        Firmware Download Support: Full, Segmented, Deferred
        Specifications Supported:
                SPC-3
                SAM-5
                SAS-3
                SPL-3
                SPC-4
                SBC-3
        Features Supported:
                Application Client Logging
                Self Test
                Automatic Write Reassignment
                Automatic Read Reassignment
                EPC
                Informational Exceptions [Mode 0]
                Translate Address
                Format Unit
                Sanitize

This entry was posted in debian, Free Software, git, Hardware, irc, linux, LSI, proliant, Seagate, storage, stretch, vacation. Bookmark the permalink.

2 Responses to The woes of 520-byte sectors

  1. https://forum.level1techs.com/t/how-to-reformat-520-byte-drives-to-512-bytes-usually/133021 says that sg_format fails because the drive is trying to prevent you from losing data, and that if you do an sg_format to 520-byte sectors followed immediately by a second sg_format to 512-byte sectors without writing any data in between, things will work.

    • C.J. Collier says:

      Hey there Marius,

      Thank you for commenting. Unfortunately, this didn’t work for me. I tried re-formatting at the 520-byte size and hoped after it was done (19 hours later) that I’d be able to format at 512. No such luck; I almost gave up after that failure.

      I was so happy to stumble upon the SeaChest tools. It was a life saver!

      Cheers,

      C.J.

Leave a Reply