NIST::NVD CWE development – follow along

I’m in the process of getting the tests passing for the 0.03 release of NIST::NVD::Store::SQLite3 wherein our hero imports the CWE data and cross-indexes it with CVEs and CPEs.

Follow along and suggest some patches. I’m developing on Debian Wheezy, but I would very much like input from devs on other platforms.

http://git.colliertech.org/?p=NIST-NVD-Store-SQLite3.git;a=summary

cjac@foxtrot:/tmp$ time git clone http://git.colliertech.org/git/NIST-NVD-Store-SQLite3.git
Cloning into 'NIST-NVD-Store-SQLite3'...

real	0m32.757s
user	0m0.200s
sys	0m0.088s
cjac@foxtrot:/tmp$ ls NIST-NVD-Store-SQLite3/t/data/
cwec_v2.1.xml  nvdcve-2.0-test.xml

Publish your patches and I’ll fetch them, or you can submit them in udiff format and I’ll review/apply. Thanks for playing along!

[edit 20120216T1456 -0800]
Seems I need to update the NIST::NVD package as well.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Can't locate object method "put_idx_cwe" via package "NIST::NVD::Update" at /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve line 77.

real	0m13.072s
user	0m12.421s
sys	0m0.044s

$ time git clone http://git.colliertech.org/git/NIST-NVD.git
Cloning into 'NIST-NVD'...

real	0m2.921s
user	0m0.016s
sys	0m0.024s

[edit 20120216T16:28 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD$ git log | head -5
commit 82c72a79ee810c2b5c269a15dca5151ad67059f9
Author: C.J. Adams-Collier 
Date:   Thu Feb 16 16:25:53 2012 -0800

    added put_idx_cwe to NIST::NVD::Update

[edit 20120216T1635 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.

real	0m13.968s
user	0m13.225s
sys	0m0.064s

Alright. before going to bed, I made those “w” characters above mean something. Same with some of the “v” characters above.

[edit 20120217T10:50 -0800]

Now processing the Categories from the CWE files. At this point we have parsed the Description elements. Next up are:

[edit 20120217T11:35 -0800]

~~Likelihood_of_Exploit~~
~~Time_of_Introduction~~
~~Affected_Resources~~
~~Applicable_Platforms~~

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................Done.
read 68 nvd entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
read 27 cwe Views
read 1 cwe Categories
read 693 cwe Weaknesses
read 1 cwe Compound Elements
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.

real	0m14.306s
user	0m13.501s
sys	0m0.072s

[edit 20120217T1333 -0800]

Alright, I’ve got enough of the CWE processing done that I feel comfortable releasing this stuff. Let’s make sure that the data got all the way down into the database. I’ll write some tests in t/cwe.t to exercise the CRUD

[edit 20120217T1345 -0800]

Tests are in place and failing (this is good, by the way). Now to turn the tests green.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ pushd ../NIST-NVD ; perl Makefile.PL ; make ; popd ; rm t/data/*.db *.db ; perl Makefile.PL ; make ; time prove -v -I../NIST-NVD/blib/lib -Iblib/lib t/cwe.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
rm: cannot remove `t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/cwe.t .. 
1..10
ok 1 - use NIST::NVD::Query;
ok 2 - no error
ok 3 - constructor returned an object of correct class
not ok 4 - cve_for_cpe returned ARRAY ref

#   Failed test 'cve_for_cpe returned ARRAY ref'
#   at t/cwe.t line 36.
#          got: ''
#     expected: 'ARRAY'
not ok 5 - cwe_for_cpe returned ARRAY ref

#   Failed test 'cwe_for_cpe returned ARRAY ref'
#   at t/cwe.t line 37.
#          got: ''
#     expected: 'ARRAY'
Can't use an undefined value as an ARRAY reference at t/cwe.t line 39.
# Looks like you planned 10 tests but ran 5.
# Looks like you failed 2 tests of 5 run.
# Looks like your test exited with 2 just after 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 7/10 subtests 

Test Summary Report
-------------------
t/cwe.t (Wstat: 512 Tests: 5 Failed: 2)
  Failed tests:  4-5
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 10 tests but ran 5.
Files=1, Tests=5,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.07 cusr  0.01 csys =  0.10 CPU)
Result: FAIL

real	0m0.442s
user	0m0.144s
sys	0m0.028s

[edit 20120220T1311 -0800]

CWE load is passing. It takes 13 seconds to process all CWEs in cwec_v2.1.xml and some recent CVEs from the 13th of December of 2011. It seems I’ve broken t/query.t, though. When it’s green I’ll push a new release. Everything is in master. Check it out if you like pain.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db ; pushd ../NIST-NVD/ ; perl Makefile.PL ; make ; popd ; perl -I../NIST-NVD/blib/lib Makefile.PL ; make ; PERL5LIB=../NIST-NVD/blib/lib:blib/lib prove -I../NIST-NVD/blib/lib -bv t/01-load-nvdcve.t 
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/01-load-nvdcve.t .. 
1..20
ok 1 - $dist_dir is a directory
ok 2 - $test_dir is a directory
ok 3 - $data_dir is a directory
ok 4 - $convert_script is a file
ok 5 - $nvd_source_file is a file
ok 6 - $db_file does not yet exist
# running /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 2>&1
ok 7 - took less than 15 seconds to load CWE data: 13
ok 8 - conversion script returned cleanly
ok 9 - database file exists
ok 10 - database file not empty
ok 11 - database file readable
ok 12 - database file writeable
ok 13 - database file not executable
ok 14 - file is correct type: [application/octet-stream]
ok 15 - $mtime is close
ok 16 - opened database file for reading
ok 17 - file contents indicate correct type: [application/octet-stream]
ok 18 - file contents indicate correct type: [application/octet-stream]
ok 19 - file contents indicate correct type: [SQLite 3.x database]
ok 20 - constructor returned an object of correct class
ok
All tests successful.
Files=1, Tests=20, 13 wallclock secs ( 0.02 usr  0.02 sys + 12.78 cusr  0.07 csys = 12.89 CPU)
Result: PASS

[edit 20120220T2220 -0800]
Okay, it’s published. I’ll need to factor the changes in to the DB_File storage engine as well.

[edit 20120225T20:27 -0800]
Just published an update. It probably works better.

Next up is creating another index from cpe urn to cwe data

Clerical Rigor

NIST::NVD CWE development – follow along

Like this:

Leave a ReplyCancel reply

NIST::NVD CWE development – follow along

Share this:

Like this:

Leave a ReplyCancel reply