NIST::NVD CWE development – follow along

I’m in the process of getting the tests passing for the 0.03 release of NIST::NVD::Store::SQLite3 wherein our hero imports the CWE data and cross-indexes it with CVEs and CPEs.

Follow along and suggest some patches. I’m developing on Debian Wheezy, but I would very much like input from devs on other platforms.

http://git.colliertech.org/?p=NIST-NVD-Store-SQLite3.git;a=summary

cjac@foxtrot:/tmp$ time git clone http://git.colliertech.org/git/NIST-NVD-Store-SQLite3.git
Cloning into 'NIST-NVD-Store-SQLite3'...

real	0m32.757s
user	0m0.200s
sys	0m0.088s
cjac@foxtrot:/tmp$ ls NIST-NVD-Store-SQLite3/t/data/
cwec_v2.1.xml  nvdcve-2.0-test.xml

Publish your patches and I’ll fetch them, or you can submit them in udiff format and I’ll review/apply. Thanks for playing along!

[edit 20120216T1456 -0800]
Seems I need to update the NIST::NVD package as well.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Can't locate object method "put_idx_cwe" via package "NIST::NVD::Update" at /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve line 77.

real	0m13.072s
user	0m12.421s
sys	0m0.044s
$ time git clone http://git.colliertech.org/git/NIST-NVD.git
Cloning into 'NIST-NVD'...

real	0m2.921s
user	0m0.016s
sys	0m0.024s

[edit 20120216T16:28 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD$ git log | head -5
commit 82c72a79ee810c2b5c269a15dca5151ad67059f9
Author: C.J. Adams-Collier 
Date:   Thu Feb 16 16:25:53 2012 -0800

    added put_idx_cwe to NIST::NVD::Update

[edit 20120216T1635 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.

real	0m13.968s
user	0m13.225s
sys	0m0.064s

Alright. before going to bed, I made those “w” characters above mean something. Same with some of the “v” characters above.

[edit 20120217T10:50 -0800]

Now processing the Categories from the CWE files. At this point we have parsed the Description elements. Next up are:

[edit 20120217T11:35 -0800]

  1. Likelihood_of_Exploit
  2. Time_of_Introduction
  3. Affected_Resources
  4. Applicable_Platforms
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................Done.
read 68 nvd entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
read 27 cwe Views
read 1 cwe Categories
read 693 cwe Weaknesses
read 1 cwe Compound Elements
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.

real	0m14.306s
user	0m13.501s
sys	0m0.072s

[edit 20120217T1333 -0800]

Alright, I’ve got enough of the CWE processing done that I feel comfortable releasing this stuff. Let’s make sure that the data got all the way down into the database. I’ll write some tests in t/cwe.t to exercise the CRUD

[edit 20120217T1345 -0800]

Tests are in place and failing (this is good, by the way). Now to turn the tests green.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ pushd ../NIST-NVD ; perl Makefile.PL ; make ; popd ; rm t/data/*.db *.db ; perl Makefile.PL ; make ; time prove -v -I../NIST-NVD/blib/lib -Iblib/lib t/cwe.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/cwe.t .. 
1..10
ok 1 - use NIST::NVD::Query;
ok 2 - no error
ok 3 - constructor returned an object of correct class
not ok 4 - cve_for_cpe returned ARRAY ref

#   Failed test 'cve_for_cpe returned ARRAY ref'
#   at t/cwe.t line 36.
#          got: ''
#     expected: 'ARRAY'
not ok 5 - cwe_for_cpe returned ARRAY ref

#   Failed test 'cwe_for_cpe returned ARRAY ref'
#   at t/cwe.t line 37.
#          got: ''
#     expected: 'ARRAY'
Can't use an undefined value as an ARRAY reference at t/cwe.t line 39.
# Looks like you planned 10 tests but ran 5.
# Looks like you failed 2 tests of 5 run.
# Looks like your test exited with 2 just after 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 7/10 subtests 

Test Summary Report
-------------------
t/cwe.t (Wstat: 512 Tests: 5 Failed: 2)
  Failed tests:  4-5
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 10 tests but ran 5.
Files=1, Tests=5,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.07 cusr  0.01 csys =  0.10 CPU)
Result: FAIL

real	0m0.442s
user	0m0.144s
sys	0m0.028s

[edit 20120220T1311 -0800]

CWE load is passing. It takes 13 seconds to process all CWEs in cwec_v2.1.xml and some recent CVEs from the 13th of December of 2011. It seems I’ve broken t/query.t, though. When it’s green I’ll push a new release. Everything is in master. Check it out if you like pain.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db ; pushd ../NIST-NVD/ ; perl Makefile.PL ; make ; popd ; perl -I../NIST-NVD/blib/lib Makefile.PL ; make ; PERL5LIB=../NIST-NVD/blib/lib:blib/lib prove -I../NIST-NVD/blib/lib -bv t/01-load-nvdcve.t 
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/01-load-nvdcve.t .. 
1..20
ok 1 - $dist_dir is a directory
ok 2 - $test_dir is a directory
ok 3 - $data_dir is a directory
ok 4 - $convert_script is a file
ok 5 - $nvd_source_file is a file
ok 6 - $db_file does not yet exist
# running /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 2>&1
ok 7 - took less than 15 seconds to load CWE data: 13
ok 8 - conversion script returned cleanly
ok 9 - database file exists
ok 10 - database file not empty
ok 11 - database file readable
ok 12 - database file writeable
ok 13 - database file not executable
ok 14 - file is correct type: [application/octet-stream]
ok 15 - $mtime is close
ok 16 - opened database file for reading
ok 17 - file contents indicate correct type: [application/octet-stream]
ok 18 - file contents indicate correct type: [application/octet-stream]
ok 19 - file contents indicate correct type: [SQLite 3.x database]
ok 20 - constructor returned an object of correct class
ok
All tests successful.
Files=1, Tests=20, 13 wallclock secs ( 0.02 usr  0.02 sys + 12.78 cusr  0.07 csys = 12.89 CPU)
Result: PASS

[edit 20120220T2220 -0800]
Okay, it’s published. I’ll need to factor the changes in to the DB_File storage engine as well.

[edit 20120225T20:27 -0800]
Just published an update. It probably works better.

Next up is creating another index from cpe urn to cwe data

This entry was posted in abuse, Databases, debian, F5 Networks, Free Software, git, Networking, NIST, perl, security, Software, SQLite, syndication, wheezy. Bookmark the permalink.

Leave a Reply