Rack coming together

I now have four six-port Qotom systems and one four-port Qotom system in my rack. I have successfully verified that the newest of these is capable of pulling about 745Mbit/s of 1500-byte frames through the switch from six other hosts. I bet it will push data at the same rate. but I can’t get iperf to distribute it over the LAG evenly. That’s another problem for another day. I know it can handle six large streams at the same time, which is great.

Posted in buster, colliertech, debian, Hardware, irc, LSI, network saturation, Networking, powerconnect | Tagged , , | Leave a comment

Recovering videos from DV tapes with Canon ZR80

I am recovering some tapes from back in the day that some of you may enjoy. Here is a log of the process so that maybe you can recover some of your own DV tapes. Seems to work well in modern Debian.

To attach to the camcorder, I used a PCI-e card that has an old firewire port and some ASIC on board. The PCI card came up and loaded the correct kernel drivers.

Here is a search link so that you can buy a similar card.

cjac@server0:~$ sudo lspci | grep 1394
b2:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEE
E 1394 OHCI Controller (rev 46)

cjac@server0:~$ sudo lsmod | grep -i firewire
firewire_ohci 45056 0
firewire_core 81920 7 firewire_ohci
crc_itu_t 16384 1 firewire_core

The dvgrab program is available on Debian under the dvgrab package.
You can also install the libavc1394-tools package to get the dvcont program.

cjac@server0:~$ sudo apt-get install dvgrab libavc1394-tools

Turn the device to “VCR” mode, attach the firewire cable and wait about five minutes. Have you watered the cat today?

cjac@server0:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
cjac@server0:~$ uname -r
cjac@server0:~$ sudo modinfo firewire_ohci | grep vermagic
vermagic: 5.2.0-0.bpo.3-amd64 SMP mod_unload modversions

cjac@server0:~$ dvcont status
Winding stopped
cjac@server0:~$ dvcont rewind
cjac@server0:~$ dvcont status
Winding reverse
cjac@server0:~$ dvcont status
Winding stopped

# make a directory to store the raw dv tape data and the
# transcodings

cjac@server0:~$ mkdir -p /srv/nfs/cj.backup/dv/oscon2006

# I’ve found that each tape stores around 12 GB of raw data, so be
# sure to perform this on a partition with tens of gigs of spare
# space

cjac@server0:~$ cd /srv/nfs/cj.backup/dv/oscon2006
cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ dvgrab –autosplit –timestamp –size 0 –rewind oscon2006-
Found AV/C device with GUID 0x0000850000e043cf
Waiting for DV…
Capture Started
“oscon2006-2006.07.26_12-37-44.dv”: 266.30 MiB 2327 frames timecode 00:01:17.26 date 2006.07.26 12:39:01
“oscon2006-2006.07.26_12-40-59.dv”: 816.76 MiB 7137 frames timecode 00:05:16.01 date 2006.07.26 12:44:57
“oscon2006-2006.07.26_12-45-06.dv”: 8420.56 MiB 73580 frames timecode 00:46:11.05 date 2006.07.26 13:26:01
“oscon2006-2006.07.26_13-32-08.dv”: 2961.27 MiB 25876 frames timecode 00:00:00.00 date 2020.06.10 10:46:25
Capture Stopped

During the capture, the dvcont status will be “Playing”:

cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ dvcont status

In a different window of the screen session or I guess a new gnome-terminal, put together a transcoding environment.

cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ sudo apt-get install libx264-155 libx264-148 ffmpeg libdatetime-format-duration-perl libdatetime-format-dateparse-perl libdatetime-perl
cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ wget https://raw.githubusercontent.com/cjac/dvscripts/master/transcode.pl && chmod u+x transcode.pl
# review transcode.pl, change $prefix

The script will detect partial transcodes and do the right thing generally, so don’t worry too much about running ./transcode.pl too often.

Results are being stored in various places including


Posted in ajax, amazon, buster, debian, family, Free Software, Hardware, html, irc, javascript, language, linux, open source, oscon 2006, perl, proliant, storage, web 2.0 | 2 Comments

The woes of 520-byte sectors

A couple years ago, I bought a 12G/s SAS disk to see if I could get it to work with the RAID controller with external SFF-8088 ports which came with the system I got while I was working at The Linux Foundation. I got an enclosure to go with it because I was enthusiastic and optimistic about my ability to get things all set up. My plan was to take the 1T disks I had in the storage server I had been using before, but which had since failed and to put them into this new enclosure. I got a couple of SFF-8088 cables, and the enclosure had some SFF-8088 to 7-pin SATA break-outs come with it. I bought an additional dual SFF-8088 to SFF-8087 adapter and a couple of SFF-8087 to 4x SFF-8482 cables. I got the 8482s because I imagined that these would be required to take full advantage of the 6TB 12G/s SAS disk mentioned above.

Unfortunately, when I attached the disks to the controller, none of them worked and I gave up frustrated and disillusioned for a good long time. Last week was spring break, and I took a few days off from work to hang out with my girls. While we weren’t all goofing off together, I was puttering around with a new HBA I got to replace the apparently non-functional HP RAID controller which came with the server.

However, when I removed the HP RAID controller and put in the LSI 9201-16E HBA I got to replace it with, there was again no SCSI love. The lsscsi command showed nothing. But the lspci command showed the controller, and I started digging through documentation to figure out what might be wrong. I eventually stumbled upon a firmware flashing ISO disk image on the Broadcom site. Broadcom apparently purchased LSI not too long ago and is now responsible for managing documentation and downloads for the legacy devices, of which I am now a proud owner.

I was able to build a USB disk from this ISO using unetbootin and emacs. I had to modify the syslinux.cfg file to correct the path and case of the filenames. And for some reason, unetbootin replaced the filename of the disk image from which freedos was supposed to boot with something useless. In any case, after corrections, the entry in syslinux.cfg looks like this:

label ubnentry20
menu label 9201_16e
append initrd=/LSI/IMG/9201_16E.IMG

Unfortunately, the firmware on the 9201_16E.IMG fat filesystem was old, so I loop mounted it and wrote a more recent version of the firmware, mptsas2.rom and sas2flsh.exe to it. I’ve uploaded it to 9201_16E.IMG in case anyone might find it useful. While I’m at it, I suppose I should put the whole usb boot image up for those who might need it. See lsi-flash-usb.img.bz2. To use this disk image, decompress the .bz2 file and dd it to your USB block device, for instance /dev/sdb in my case:

$ bunzip2 /tmp/lsi-flash-usb.img.bz2 | sudo dd of=/dev/sdb

My Proliant DL370 G6 server doesn’t have a fancy new UEFI system, so I used this boot disk to get access to the SAS controller in real mode. Once I booted it and selected 9201_16e (the 21st option from the top), I had to rapidly press the up or down arrow and select the EMM386 option to load the driver. For those playing along at home, I placed the more recent LSI controller firmware and BIOS in the \CJ directory, which has the same structure as the \LSI directory from the original disk image. You should be able to

cd \CJ

and run


Answer the prompts, and afterwards you should have a controller with P20 firmware and BIOS written to it.

Daunting as this was, it only got me most of the way through the process. All of my SATA disks were working at this point, but the SAS disk was not. After some sleuthing, I discovered that the disk was SCSI formatted to 520 byte sectors. Linux does not accept sector sizes that are not a power of 2, and so i was unable to do anything with the disk using fdisk, kpartx, etc. Research pointed me to the sg_format utility, but I was unable to make that work. Here’s the output of the command:

$ sudo sg_format --format --size 512 -vv /dev/sg4
open /dev/sg4 with flags=0x802
    inquiry cdb: 12 00 00 00 24 00 
    SEAGATE   DKS2F-H6R0SS      7FA6   peripheral_type: disk [0x0]
    inquiry cdb: 12 01 00 00 24 00 
    inquiry cdb: 12 01 80 01 00 00 
      Unit serial number: Z4D0M2BF0000W515S4WH
    inquiry cdb: 12 01 83 01 00 00 
      LU name: 5000c50062ba7973
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 28 bytes
Mode Sense (block descriptor) data, prior to changes:
Mode sense number of blocks maxed out, set longlba
    mode sense (10) cdb: 5a 10 01 00 00 00 00 00 fc 00 
    mode sense (10): pass-through requested 252 bytes (data-in) but got 36 bytes
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=11473076960 [0x2abd942e0]
  Block size=520 [0x208]
    mode select (10) cdb: 55 11 00 00 00 00 00 00 22 00 
    mode select (10) parameter list
00 00 00 00 01 00 00 10  00 00 00 00 00 00 00 00
00 00 00 00 00 00 04 00  01 0a 0c 14 ff 00 00 00
05 00
mode select (10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
  Descriptor type: Sense key specific: Field pointer:
        Error in Command: byte 7 bit 7
  Descriptor type: Field replaceable unit code: 0x5
  Descriptor type: Vendor specific [0x80]
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 
 Raw sense data (in hex):
        72 05 1a 00 00 00 00 1c  02 06 00 00 cf 00 07 00
        03 02 00 05 80 0e 00 00  00 00 00 00 00 00 00 00
        00 00 00 00
MODE SELECT command: Illegal request sense key, apart from Invalid opcode

I was about to give up, but for some reason did not. I tried to find firmware and read any documentation about my disk, a Seagate ST6000NM0024. Eventually I found a github repo called ToolBin. In this repo there is a program named SeaChest_Format_121_1183_64 which seems to use a proprietary SCSI format command to tell the drive to format itself using 512-byte sectors. I can’t say for certain yet that it works, but it did not immediately fail as sg_format did. The command I used (and this will vary for you, depending on what sg_scan -i tells you is your scsi generic device) is:

./SeaChest_Format_121_1183_64 -d /dev/sg5 --fastFormat 1 --formatUnit 512 --confirm I-understand-this-command-will-erase-all-data-on-the-drive

Assuming this command results in success after the 12 hour run, I will purchase another 7 of these drives and fill out my disk array!

Sun Apr 14 13:39:48 PDT 2019
        Format Unit Progress = 3.61%

Thanks to the #linux-raid channel for helping me through this!


Mon Apr 15 08:16:00 PDT 2019
        Format Unit Progress = 99.97%
Mon Apr 15 08:17:00 PDT 2019
        Format Unit Progress = 100.00%

[70874.677389] sd 3:0:2:0: device_block, handle(0x0012)
[70876.176199] sd 3:0:2:0: device_unblock and setting to running, handle(0x0012)
[70876.249541] mpt2sas_cm0: removing handle(0x0012), sas_addr(0x5000c50062ba7971)
[70876.253344] mpt2sas_cm0: removing : enclosure logical id(0x500062b20059ecc0), slot(1)
[70886.680412] scsi 3:0:3:0: Direct-Access     SEAGATE  DKS2F-H6R0SS     7FA6 PQ: 0 ANSI: 5
[70886.684166] scsi 3:0:3:0: SSP: handle(0x0012), sas_addr(0x5000c50062ba7971), phy(2), device_name(0x5000c50062ba7970)
[70886.688833] scsi 3:0:3:0: SSP: enclosure_logical_id(0x500062b20059ecc0), slot(1)
[70886.750049] sd 3:0:3:0: Attached scsi generic sg5 type 0
[70886.750143] sd 3:0:3:0: [sdd] Spinning up disk...
[70887.765498] .
[70888.789522] .
[70889.813591] .
[70890.837718] .
[70891.861715] .
[70892.885876] .
[70893.909889] .
[70894.933912] .
[70895.957989] .
[70896.982028] .
[70898.006133] .
[70899.030215] .
[70900.054269] .
[70901.078340] .
[70902.102359] .
[70903.478491] .
[70904.502557] .
[70906.823608] ready
[70906.826165] sd 3:0:3:0: [sdd] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[70906.829779] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[70906.834708] sd 3:0:3:0: [sdd] Write Protect is off
[70906.837364] sd 3:0:3:0: [sdd] Mode Sense: 13 00 10 08
[70906.838142] sd 3:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[70906.867106] sd 3:0:3:0: [sdd] Attached SCSI disk

$ sudo ./SeaChest_Info_141_1183_64  -d /dev/sg5 -i
 SeaChest_Info - Seagate drive utilities - NVMe Enabled
 Copyright (c) 2014-2018 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 SeaChest_Info Version: 1.4.1-1_18_3 X86_64
 Build Date: Oct 18 2018
 Today: Mon Apr 15 14:52:33 2019

/dev/sg5 - DKS2F-H6R0SS - Z4D0M2BF0000W515S4WH - SCSI
        Vendor ID: SEAGATE 
        Model Number: DKS2F-H6R0SS    
        Serial Number: Z4D0M2BF0000W515S4WH
        Firmware Revision: 7FA6
        World Wide Name: 5000C50062BA7973
        Copyright: Copyright (c) 2014 Seagate All rights reserved 
        Drive Capacity (TB/TiB): 6.00/5.46
        Temperature Data:
                Current Temperature (C): 45
                Highest Temperature (C): Not Reported
                Lowest Temperature (C): Not Reported
        Power On Time:  21 days 11 hours 8 minutes 
        Power On Hours: 515.13
        MaxLBA: 11721045167
        Native MaxLBA: Not Reported
        Logical Sector Size (B): 512
        Physical Sector Size (B): 4096
        Sector Alignment: 0
        Rotation Rate (RPM): 7200
        Form Factor (inch): 3.5
        Last DST information:
                DST has never been run
        Long Drive Self Test Time:  10 hours 2 minutes 
        Interface speed:
                Port 0 (Current Port)
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): 6.0
                Port 1
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): Not Reported
        Annualized Workload Rate (TB/yr): 0.00
        Total Bytes Read (MB): 6.39
        Total Bytes Written (MB): 14.99
        Encryption Support: Not Supported
        Cache Size (MiB): Not Reported
        Read Look-Ahead: Not Supported
        Write Cache: Disabled
        SMART Status: Good
        ATA Security Information: Not Supported
        Firmware Download Support: Full, Segmented, Deferred
        Specifications Supported:
        Features Supported:
                Application Client Logging
                Self Test
                Automatic Write Reassignment
                Automatic Read Reassignment
                Informational Exceptions [Mode 0]
                Translate Address
                Format Unit

Posted in debian, Free Software, git, Hardware, irc, linux, LSI, proliant, Seagate, storage, stretch, vacation | 6 Comments

I’m running an ethereum node

cjac@server0:~/Downloads/geth-linux-amd64-1.8.14-316fc7ec$ df -h ~/.ethereum/
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg00-ethereum  148G  130G   11G  93% /home/cjac/.ethereum

this was from before I ran

./geth --syncmode "fast" --cache=1024
Posted in C.J. Insider, debian, linux, open source, proliant, storage, stretch, Uncategorized | Leave a comment

Using SonarQube 5.4, Maven 3.3.9, Jenkins 2.19.1 on systems with both Java 1.7 and 1.8

Hello folks! My team spent hours and hours beating our head against a Sonar deployment problem on Ubuntu Trusty (14.04 LTS). I thought I might share our findings so that you won’t have to!

As you probably know, Trusty only makes Java Development Kit 1.7 available on the stock installation. The current stable version of the Java is 1.8. The way we install this is to use the OpenJDK PPA, generously uploaded by our dear friend Matthias Klose.

To make things even more exciting, a modern Maven is not available on this platform. And so we use the stock Maven 3.3.9 tarball distribution. This tarball distribution does not integrate well with Debian, and so, when we tell the system using sudo update-java-alternatives -s /usr/lib/jvm/java-1.8.0-openjdk-amd64 that we wish to use Java 1.8 as our default system JDK, it does not get the message.

The only way to reliably let Maven know which java you wish to use is to set the JAVA_HOME environment variable. In order to do this within the Jenkins environment, one must select the JDK one wishes to use:


To make things worse, this option is not, as one might expect, available for editing in a stock Jenkins 2.x installation. In Jenkins 1.x, one would be able to specify which java one wished to use just by specifying “openjdk8” in a field. With Jenkins 2.x, the field does not exist unless a configuration option in an unrelated form is set.

So! One should first select Manage Jenkins -> Global Tool Configuration:


Once this form is open, look for the “JDK installations…” button:


Click it very thoroughly just once.

You’ll be presented with a form into which you may enter details about the various JDKs your build executors may have access to. You’ll refer to them in your job configuration by the value of their “Name” field, and when executing the build, Jenkins will set JAVA_HOME to the value of the (you guessed it) JAVA_HOME field:


Once these entries are made, they can be selected in two place.

1) on the ZMQ Event Publisher:


2) in the post-build actions under SonarQube analysis with Maven (advanced)


And that’s how it’s done!

Here’s the details from my colleague, Thanh:


Posted in debian, Free Software, java, linux, Linux Foundation, open source, openstack, Software, ubuntu, virtualization, work | Leave a comment

OpenDaylight Symposium 2016

I’ll write this later.  Keywords and notes inline












Open Source




OSI stack


Developers and users working together.

Fast Dev/Test cycles

Sessions start at 10:30

Message bus

Minimal install needs to be smaller

Most functionalities must be implemented in modules

Upgrade path complexity

Out of band control channel.
OPNFV customers don’t run stock releases, make customisations and plugins easier.

Message bus

Distribution vendors should not be expected to perform package maintenance.  Get the distribution ready for fast pathing in to backports, security, testing  and unstable.

Posted in android, Azure, centos, colliertech, customer experience, debian, dns, ESXi, Free Software, freenode, irc, jessie, kvm, libvirt, linux, lvm, microsoft, network saturation, Networking, open source, qemu, quagga, rate limiting, Software, Telephony, traffic shaping, ubuntu, VBox, virtualization, washington, wheezy, work, xen | Leave a comment

virt manager cannot find suitable emulator for x86 64

Looks like I was missing qemu-kvm.

$ sudo apt-get install qemu-kvm qemu-system

Posted in debian, Free Software, kvm, libvirt, linux, open source, qemu, ubuntu, virtualization | Leave a comment

Some work on a VyOS image with Let’s Encrypt certs

I put some packages together this weekend. It’s been a while since I’ve debuilt anything officially.

The plan is to build a binding to the libgnutls.so.30 API. The certtool CSR (REQ) generation interface does not allow me to create a CRL with “not critical” attributes set on purposes. Maybe if I do it a bit closer to the metal it will be easier…

Posted in 2016, colliertech, cryptography, debian, Free Software, git, gnutls, gpl, irc, kvm, libvirt, linux, Networking, perl, proliant, qemu, security, sid, Software, virtualization, vyos, washington | Leave a comment

OMG Maven 3.0.4 on stretch

“Why?”, you might ask, would one want to run something other than the most recent version of Maven on the very newest and fangledest breed of the linux distribution we have all loved for so long.

“Because!”, I might answer, I’m trying to get the nexus-apt-plugin working on nexus.fd.io, and the version of nexus we’re running there explained to me in quite uncertain terms that it would talk to no other version of maven than 3.0.4 or something else that is not packaged for debian.

So I grabbed the source for version 3.0.4 from wheezy and patched it up to work with stretch:

$ cd /usr/src/deb
$ dget http.debian.net/debian/pool/main/m/maven/maven_3.0.4-3+deb7u1.dsc
$ cd maven-3.0.4
$ perl -i -pe 's/(libmodello-maven-plugin)1.4(-java)/$1$2/' debian/control
$ quilt pop -a
$ quild push 1
$ perl -i -pe 's/-1.4.x\.jar/.jar/' build.xml
$ perl -i -pe 's/google-collections/guava/' build.xml
$ perl -i -pe 's/\s+$//' build.xml
$ quilt refresh
$ quilt pop
$ quilt push -a
$ debuild -uc -us
$ sudo apt-get remove maven libmaven3-core-java
$ sudo dpkg -i ../maven_3.0.4-3+deb7u1_all.deb

And now I can build the silly nexus-apt-plugin…

$ mkdir -p /usr/src/git/github
$ git clone git@github.com:LLC-Technologies-Collier/nexus-apt-plugin.git /usr/src/git/github/nexus-apt-plugin
$ cd /usr/src/git/github/nexus-apt-plugin
$ mvn compile && mvn -q test
Posted in CLI, compiler, debian, Free Software, Linux Foundation, Nexus, sid, Software, stretch, wheezy, work | Leave a comment

The Provisioning Act of 2016


Posted in Uncategorized | Leave a comment