The woes of 520-byte sectors

A couple years ago, I bought a 12G/s SAS disk to see if I could get it to work with the RAID controller with external SFF-8088 ports which came with the system I got while I was working at The Linux Foundation. I got an enclosure to go with it because I was enthusiastic and optimistic about my ability to get things all set up. My plan was to take the 1T disks I had in the storage server I had been using before, but which had since failed and to put them into this new enclosure. I got a couple of SFF-8088 cables, and the enclosure had some SFF-8088 to 7-pin SATA break-outs come with it. I bought an additional dual SFF-8088 to SFF-8087 adapter and a couple of SFF-8087 to 4x SFF-8482 cables. I got the 8482s because I imagined that these would be required to take full advantage of the 6TB 12G/s SAS disk mentioned above.

Unfortunately, when I attached the disks to the controller, none of them worked and I gave up frustrated and disillusioned for a good long time. Last week was spring break, and I took a few days off from work to hang out with my girls. While we weren’t all goofing off together, I was puttering around with a new HBA I got to replace the apparently non-functional HP RAID controller which came with the server.

However, when I removed the HP RAID controller and put in the LSI 9201-16E HBA I got to replace it with, there was again no SCSI love. The lsscsi command showed nothing. But the lspci command showed the controller, and I started digging through documentation to figure out what might be wrong. I eventually stumbled upon a firmware flashing ISO disk image on the Broadcom site. Broadcom apparently purchased LSI not too long ago and is now responsible for managing documentation and downloads for the legacy devices, of which I am now a proud owner.

I was able to build a USB disk from this ISO using unetbootin and emacs. I had to modify the syslinux.cfg file to correct the path and case of the filenames. And for some reason, unetbootin replaced the filename of the disk image from which freedos was supposed to boot with something useless. In any case, after corrections, the entry in syslinux.cfg looks like this:

label ubnentry20
menu label 9201_16e
kernel /BOOT/ISOLINUX/MEMDISK
append initrd=/LSI/IMG/9201_16E.IMG

Unfortunately, the firmware on the 9201_16E.IMG fat filesystem was old, so I loop mounted it and wrote a more recent version of the firmware, mptsas2.rom and sas2flsh.exe to it. I’ve uploaded it to 9201_16E.IMG in case anyone might find it useful. While I’m at it, I suppose I should put the whole usb boot image up for those who might need it. See lsi-flash-usb.img.bz2. To use this disk image, decompress the .bz2 file and dd it to your USB block device, for instance /dev/sdb in my case:

$ bunzip2 /tmp/lsi-flash-usb.img.bz2 | sudo dd of=/dev/sdb

My Proliant DL370 G6 server doesn’t have a fancy new UEFI system, so I used this boot disk to get access to the SAS controller in real mode. Once I booted it and selected 9201_16e (the 21st option from the top), I had to rapidly press the up or down arrow and select the EMM386 option to load the driver. For those playing along at home, I placed the more recent LSI controller firmware and BIOS in the \CJ directory, which has the same structure as the \LSI directory from the original disk image. You should be able to

cd \CJ

and run

9201.bat

Answer the prompts, and afterwards you should have a controller with P20 firmware and BIOS written to it.

Daunting as this was, it only got me most of the way through the process. All of my SATA disks were working at this point, but the SAS disk was not. After some sleuthing, I discovered that the disk was SCSI formatted to 520 byte sectors. Linux does not accept sector sizes that are not a power of 2, and so i was unable to do anything with the disk using fdisk, kpartx, etc. Research pointed me to the sg_format utility, but I was unable to make that work. Here’s the output of the command:

$ sudo sg_format --format --size 512 -vv /dev/sg4
open /dev/sg4 with flags=0x802
    inquiry cdb: 12 00 00 00 24 00 
    SEAGATE   DKS2F-H6R0SS      7FA6   peripheral_type: disk [0x0]
      PROTECT=0
    inquiry cdb: 12 01 00 00 24 00 
    inquiry cdb: 12 01 80 01 00 00 
      Unit serial number: Z4D0M2BF0000W515S4WH
    inquiry cdb: 12 01 83 01 00 00 
      LU name: 5000c50062ba7973
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 28 bytes
Mode Sense (block descriptor) data, prior to changes:
Mode sense number of blocks maxed out, set longlba
    mode sense (10) cdb: 5a 10 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 36 bytes
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=11473076960 [0x2abd942e0]
  Block size=520 [0x208]
    mode select (10) cdb: 55 11 00 00 00 00 00 00 22 00 
    mode select (10) parameter list
00 00 00 00 01 00 00 10  00 00 00 00 00 00 00 00
00 00 00 00 00 00 04 00  01 0a 0c 14 ff 00 00 00
05 00
mode select (10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
  Descriptor type: Sense key specific: Field pointer:
        Error in Command: byte 7 bit 7
  Descriptor type: Field replaceable unit code: 0x5
  Descriptor type: Vendor specific [0x80]
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 
 Raw sense data (in hex):
        72 05 1a 00 00 00 00 1c  02 06 00 00 cf 00 07 00
        03 02 00 05 80 0e 00 00  00 00 00 00 00 00 00 00
        00 00 00 00
MODE SELECT command: Illegal request sense key, apart from Invalid opcode

I was about to give up, but for some reason did not. I tried to find firmware and read any documentation about my disk, a Seagate ST6000NM0024. Eventually I found a github repo called ToolBin. In this repo there is a program named SeaChest_Format_121_1183_64 which seems to use a proprietary SCSI format command to tell the drive to format itself using 512-byte sectors. I can’t say for certain yet that it works, but it did not immediately fail as sg_format did. The command I used (and this will vary for you, depending on what sg_scan -i tells you is your scsi generic device) is:

./SeaChest_Format_121_1183_64 -d /dev/sg5 --fastFormat 1 --formatUnit 512 --confirm I-understand-this-command-will-erase-all-data-on-the-drive

Assuming this command results in success after the 12 hour run, I will purchase another 7 of these drives and fill out my disk array!

Sun Apr 14 13:39:48 PDT 2019
        Format Unit Progress = 3.61%

Thanks to the #linux-raid channel for helping me through this!

Update:

Mon Apr 15 08:16:00 PDT 2019
        Format Unit Progress = 99.97%
Mon Apr 15 08:17:00 PDT 2019
        Format Unit Progress = 100.00%

[70874.677389] sd 3:0:2:0: device_block, handle(0x0012)
[70876.176199] sd 3:0:2:0: device_unblock and setting to running, handle(0x0012)
[70876.249541] mpt2sas_cm0: removing handle(0x0012), sas_addr(0x5000c50062ba7971)
[70876.253344] mpt2sas_cm0: removing : enclosure logical id(0x500062b20059ecc0), slot(1)
[70886.680412] scsi 3:0:3:0: Direct-Access     SEAGATE  DKS2F-H6R0SS     7FA6 PQ: 0 ANSI: 5
[70886.684166] scsi 3:0:3:0: SSP: handle(0x0012), sas_addr(0x5000c50062ba7971), phy(2), device_name(0x5000c50062ba7970)
[70886.688833] scsi 3:0:3:0: SSP: enclosure_logical_id(0x500062b20059ecc0), slot(1)
[70886.750049] sd 3:0:3:0: Attached scsi generic sg5 type 0
[70886.750143] sd 3:0:3:0: [sdd] Spinning up disk...
[70887.765498] .
[70888.789522] .
[70889.813591] .
[70890.837718] .
[70891.861715] .
[70892.885876] .
[70893.909889] .
[70894.933912] .
[70895.957989] .
[70896.982028] .
[70898.006133] .
[70899.030215] .
[70900.054269] .
[70901.078340] .
[70902.102359] .
[70903.478491] .
[70904.502557] .
[70906.823608] ready
[70906.826165] sd 3:0:3:0: [sdd] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[70906.829779] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[70906.834708] sd 3:0:3:0: [sdd] Write Protect is off
[70906.837364] sd 3:0:3:0: [sdd] Mode Sense: 13 00 10 08
[70906.838142] sd 3:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[70906.867106] sd 3:0:3:0: [sdd] Attached SCSI disk

$ sudo ./SeaChest_Info_141_1183_64  -d /dev/sg5 -i
==========================================================================================
 SeaChest_Info - Seagate drive utilities - NVMe Enabled
 Copyright (c) 2014-2018 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 SeaChest_Info Version: 1.4.1-1_18_3 X86_64
 Build Date: Oct 18 2018
 Today: Mon Apr 15 14:52:33 2019
==========================================================================================

/dev/sg5 - DKS2F-H6R0SS - Z4D0M2BF0000W515S4WH - SCSI
        Vendor ID: SEAGATE 
        Model Number: DKS2F-H6R0SS    
        Serial Number: Z4D0M2BF0000W515S4WH
        Firmware Revision: 7FA6
        World Wide Name: 5000C50062BA7973
        Copyright: Copyright (c) 2014 Seagate All rights reserved 
        Drive Capacity (TB/TiB): 6.00/5.46
        Temperature Data:
                Current Temperature (C): 45
                Highest Temperature (C): Not Reported
                Lowest Temperature (C): Not Reported
        Power On Time:  21 days 11 hours 8 minutes 
        Power On Hours: 515.13
        MaxLBA: 11721045167
        Native MaxLBA: Not Reported
        Logical Sector Size (B): 512
        Physical Sector Size (B): 4096
        Sector Alignment: 0
        Rotation Rate (RPM): 7200
        Form Factor (inch): 3.5
        Last DST information:
                DST has never been run
        Long Drive Self Test Time:  10 hours 2 minutes 
        Interface speed:
                Port 0 (Current Port)
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): 6.0
                Port 1
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): Not Reported
        Annualized Workload Rate (TB/yr): 0.00
        Total Bytes Read (MB): 6.39
        Total Bytes Written (MB): 14.99
        Encryption Support: Not Supported
        Cache Size (MiB): Not Reported
        Read Look-Ahead: Not Supported
        Write Cache: Disabled
        SMART Status: Good
        ATA Security Information: Not Supported
        Firmware Download Support: Full, Segmented, Deferred
        Specifications Supported:
                SPC-3
                SAM-5
                SAS-3
                SPL-3
                SPC-4
                SBC-3
        Features Supported:
                Application Client Logging
                Self Test
                Automatic Write Reassignment
                Automatic Read Reassignment
                EPC
                Informational Exceptions [Mode 0]
                Translate Address
                Format Unit
                Sanitize

Posted in debian, Free Software, git, Hardware, irc, linux, LSI, proliant, Seagate, storage, stretch, vacation | 3 Comments

I’m running an ethereum node

cjac@server0:~/Downloads/geth-linux-amd64-1.8.14-316fc7ec$ df -h ~/.ethereum/
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg00-ethereum  148G  130G   11G  93% /home/cjac/.ethereum

this was from before I ran

./geth --syncmode "fast" --cache=1024
Posted in C.J. Insider, debian, linux, open source, proliant, storage, stretch, Uncategorized | Leave a comment

Using SonarQube 5.4, Maven 3.3.9, Jenkins 2.19.1 on systems with both Java 1.7 and 1.8

Hello folks! My team spent hours and hours beating our head against a Sonar deployment problem on Ubuntu Trusty (14.04 LTS). I thought I might share our findings so that you won’t have to!

As you probably know, Trusty only makes Java Development Kit 1.7 available on the stock installation. The current stable version of the Java is 1.8. The way we install this is to use the OpenJDK PPA, generously uploaded by our dear friend Matthias Klose.

To make things even more exciting, a modern Maven is not available on this platform. And so we use the stock Maven 3.3.9 tarball distribution. This tarball distribution does not integrate well with Debian, and so, when we tell the system using sudo update-java-alternatives -s /usr/lib/jvm/java-1.8.0-openjdk-amd64 that we wish to use Java 1.8 as our default system JDK, it does not get the message.

The only way to reliably let Maven know which java you wish to use is to set the JAVA_HOME environment variable. In order to do this within the Jenkins environment, one must select the JDK one wishes to use:

openjdk8-ubuntu1404-as-jdk

To make things worse, this option is not, as one might expect, available for editing in a stock Jenkins 2.x installation. In Jenkins 1.x, one would be able to specify which java one wished to use just by specifying “openjdk8” in a field. With Jenkins 2.x, the field does not exist unless a configuration option in an unrelated form is set.

So! One should first select Manage Jenkins -> Global Tool Configuration:

Jenkins2-Global_Tool_Configuration

Once this form is open, look for the “JDK installations…” button:

Jenkins2-JDK_installations

Click it very thoroughly just once.

You’ll be presented with a form into which you may enter details about the various JDKs your build executors may have access to. You’ll refer to them in your job configuration by the value of their “Name” field, and when executing the build, Jenkins will set JAVA_HOME to the value of the (you guessed it) JAVA_HOME field:

Jenkins2-JDK_installations-expanded

Once these entries are made, they can be selected in two place.

1) on the ZMQ Event Publisher:

jdk-select-project

2) in the post-build actions under SonarQube analysis with Maven (advanced)

postbuild-sonarqube-select-jdk

And that’s how it’s done!

Here’s the details from my colleague, Thanh:

https://lists.fd.io/pipermail/honeycomb-dev/2016-October/000387.html

Posted in debian, Free Software, java, linux, Linux Foundation, open source, openstack, Software, ubuntu, virtualization, work | Leave a comment

OpenDaylight Symposium 2016

I’ll write this later.  Keywords and notes inline

Cloud

5G

AT&T

Ericsson

SDN

FD.io

Cisco

Intel

Linux

Debian

Ubuntu

Open Source

Hydrogen

Helium

Lithium

OSI stack

Usability

Developers and users working together.

Fast Dev/Test cycles

Sessions start at 10:30

Message bus

Minimal install needs to be smaller

Most functionalities must be implemented in modules

Upgrade path complexity

Out of band control channel.
OPNFV customers don’t run stock releases, make customisations and plugins easier.

Message bus

Distribution vendors should not be expected to perform package maintenance.  Get the distribution ready for fast pathing in to backports, security, testing  and unstable.

Posted in android, Azure, centos, colliertech, customer experience, debian, dns, ESXi, Free Software, freenode, irc, jessie, kvm, libvirt, linux, lvm, microsoft, network saturation, Networking, open source, qemu, quagga, rate limiting, Software, Telephony, traffic shaping, ubuntu, VBox, virtualization, washington, wheezy, work, xen | Leave a comment

virt manager cannot find suitable emulator for x86 64

Looks like I was missing qemu-kvm.

$ sudo apt-get install qemu-kvm qemu-system

Posted in debian, Free Software, kvm, libvirt, linux, open source, qemu, ubuntu, virtualization | Leave a comment

Some work on a VyOS image with Let’s Encrypt certs

I put some packages together this weekend. It’s been a while since I’ve debuilt anything officially.

The plan is to build a binding to the libgnutls.so.30 API. The certtool CSR (REQ) generation interface does not allow me to create a CRL with “not critical” attributes set on purposes. Maybe if I do it a bit closer to the metal it will be easier…

Posted in 2016, colliertech, cryptography, debian, Free Software, git, gnutls, gpl, irc, kvm, libvirt, linux, Networking, perl, proliant, qemu, security, sid, Software, virtualization, vyos, washington | Leave a comment

OMG Maven 3.0.4 on stretch

“Why?”, you might ask, would one want to run something other than the most recent version of Maven on the very newest and fangledest breed of the linux distribution we have all loved for so long.

“Because!”, I might answer, I’m trying to get the nexus-apt-plugin working on nexus.fd.io, and the version of nexus we’re running there explained to me in quite uncertain terms that it would talk to no other version of maven than 3.0.4 or something else that is not packaged for debian.

So I grabbed the source for version 3.0.4 from wheezy and patched it up to work with stretch:

$ cd /usr/src/deb
$ dget http.debian.net/debian/pool/main/m/maven/maven_3.0.4-3+deb7u1.dsc
$ cd maven-3.0.4
$ perl -i -pe 's/(libmodello-maven-plugin)1.4(-java)/$1$2/' debian/control
$ quilt pop -a
$ quild push 1
$ perl -i -pe 's/-1.4.x\.jar/.jar/' build.xml
$ perl -i -pe 's/google-collections/guava/' build.xml
$ perl -i -pe 's/\s+$//' build.xml
$ quilt refresh
$ quilt pop
$ quilt push -a
$ debuild -uc -us
$ sudo apt-get remove maven libmaven3-core-java
$ sudo dpkg -i ../maven_3.0.4-3+deb7u1_all.deb

And now I can build the silly nexus-apt-plugin…

$ mkdir -p /usr/src/git/github
$ git clone git@github.com:LLC-Technologies-Collier/nexus-apt-plugin.git /usr/src/git/github/nexus-apt-plugin
$ cd /usr/src/git/github/nexus-apt-plugin
$ mvn compile && mvn -q test
Posted in CLI, compiler, debian, Free Software, Linux Foundation, Nexus, sid, Software, stretch, wheezy, work | Leave a comment

The Provisioning Act of 2016

image

Posted in Uncategorized | Leave a comment

My new rig

Go ahead and drool.

workie-2016.pdf

Posted in Uncategorized | Leave a comment

LLC-Technologies-Collier/Demo-SCCC-Byte-AngularJS

Hello dear readers and attendees,

This is the post that I will be/ will have been referencing during my presentation to the Seattle Central Community College’s Byte club on Thursday, December 10th at 1500-1630.

I will begin with a bit of an autobio and find out what kind of students we have in attendance. Please feel free to comment if you’d like to keep in touch before or after the presentation. I will discuss some of the bits and pieces of some industry standard platforms which I’ve developed, deployed, maintained, managed, co-operated, administered and replaced. We can discuss some of the patterns that work well in the industry, and some that are a bit harder to tame.

Once we have touched most of the areas of specialization represented at the meeting, I will dive in to an AngularJS demo I am developing in github here:

https://github.com/LLC-Technologies-Collier/Demo-SCCC-Byte-AngularJS/tree/master

To follow along with the presentation, please run these commands or something similar. My development environment is Debian stable. So yes, this means that we’re not doing a demo of the state of the art. But it also means that the infrastructure has been exercised under load and in production.

Install Debian package dependencies

First, install the debian packages of nodejs and npm, the node package manager:

cjac@debian0:~$ sudo apt-get install nodejs nodejs-dev nodejs-legacy npm

Check out the git repository

After this, check out the repository from github and create a branch for your work:

cjac@debian0:~$ mkdir -p /usr/src/git/github/LLC-Technologies-Collier
cjac@debian0:~$ cd /usr/src/git/github/LLC-Technologies-Collier
cjac@debian0:/usr/src/git/github/LLC-Technologies-Collier$ git clone git@github.com:LLC-Technologies-Collier/Demo-SCCC-Byte-AngularJS.git
...
cjac@debian0:/usr/src/git/github/LLC-Technologies-Collier$ cd Demo-SCCC-Byte-AngularJS
cjac@debian0:.../Demo-SCCC-Byte-AngularJS$ git checkout -b $USER

Upgrade to latest npm, install deps

Once we have the git repository checked out, we’ll grab the latest version of npm and the rest of the node modules

cjac@debian0:.../Demo-SCCC-Byte-AngularJS$ export PATH="$PWD/node_modules/.bin:$PATH"
cjac@debian0:.../Demo-SCCC-Byte-AngularJS$ npm install --save-exact npm@"2.1.0"
cjac@debian0:.../Demo-SCCC-Byte-AngularJS$ npm install --save-exact `cat pkgackage-list.txt`

This post and the associated git repository will be updated between now and the presentation on Thursday. Please chime in and feel free to get involved!

C.J.

Posted in 2015, ajax, C.J. Insider, colliertech, cryptography, css, Databases, debian, Free Software, freenode, git, Hardware, html, irc, javascript, jessie, linux, mail, microsoft, mysql, Networking, open source, perl, security, Software, Telephony, ubuntu, VBox, virtualization, washington, wiki, winter, x509 | 3 Comments