Home

Python RPM Specs Needing a Loving Home @ 2010-06-22 16:37:30.688423
Filed under: Linux  Code  Tech  Fedora  Python 
I found myself in need of some (currently) unpackaged python libraries and tools so I decided to do some quick packaging. If you are using these libraries (or want to) on a Fedora, RHEL or CentOS system and want to start packaging in Fedora then one of these could be an easy jumping in package. Note that they are not perfect so they still need a bit of love before getting them approved but it's (slightly) easier than starting from scratch!

django-picklefield.spec: provides an implementation of a pickled object field
python-amqplib.spec: Client library for AMQP
python-anyjson.spec: Wraps the best available JSON implementation available in a common interface
python-billiard.spec: Multiprocessing Pool Extensions
python-carrot.spec: AMQP Messaging Framework for Python
python-importlib.spec: Backport of importlib.import_module() from Python 2.7
python-celery.spec: task queue/job queue based on distributed message passing (requires all of the above)

 digg it   seed it   del.icio.us   ma.gnolia
Comments: 0 Tags:          


Changelog Format @ 2010-06-14 22:06:54.198506
Filed under: Code  Tech  Frustration  Fedora 
This has been bugging me fore a while. Many projects and products have a changelog. It's great! We can see in a file what changes have been made from release 1 to release 2. Wouldn't it be great to watch an upstream changelog file and use something like Buildbot that would trigger a build on a new release. Then my brain started working. It told me that humans are much better at parsing information provided in different textual formats or markups. Let me explain ....

Let's take a look at the victims project. Even though victims doesn't have a special changelog, we will follow the general scm changelog information. Victims has a changelog like so:

2010-05-20  Steve 'Ashcrow' Milner  

        * setup.py:
        added archivers module to the setup script
        [4cd8f0133b44] [tip]

2010-05-18  Steve 'Ashcrow' Milner  

        * README, src/victims/__init__.py, src/victims/archivers/__init__.py:
        rpm is now listed as a useable archive closing #8
        [e71ad437f9f4]


Based off this information we can easily create a parser! We care about the date, author/email, description and the release (tag). Through the magic of a little bit of regex the following works decent enough ... 

(\d{4}-\d{2}-\d{2})  (.*)  <(.*)>\n\n.*:\n[ ]*(.*)\n[ ]*(.*)


Now we can parse changelogs! Yay! Oh, but then our brain explodes in fear since this is not the only project out there. Surely everyone uses the same format! Let's use nmap as a second project example.

# Nmap Changelog ($Id: CHANGELOG 18109 2010-06-14 18:48:07Z drazen $); -*-text-*-

o [NSE] Added additional vulnerability checks to smb-check-vulns.nse. These checks
  are intrusive and have MS06-025, MS07-029 designations.
  
o [NSE] Added dns-cache-snoop.nse by Eugene Alexeev. This script does
  cache snooping by either sending non-recursive queries or by measuring
  response times.


Well that isn't so bad! With a little regex we could ... wait ... if I have to do this twice with two different projects am I going to need to do this many, many more times before I create Skynet^H^H^H^H^H^Han uber parser smart enough to figure out what accent, dialect, markup, etc.. a changelog may be in? It sure seems that way!

This is when another thought entered by brain (TWO IN ONE DAY!!!): Surely someone else has thought of this. There must be a commonly used format that shares this information for easy inclusion. As it turns out, I could only find one format for this and it doesn't exactly match. The project I'm talking about is doap. While the project does seem interesting, it seems to focus more so on information about a project and it's services and not so much about project releases and changes that have happened between those releases.

Long story long .... am I out of luck? Is there not a format in the works to deal with release information such as this in an open way? If there really isn't, is anyone interested in creating a format? It seems to me that this would be quite useful for package maintainers, system administrators and developers. Hit me up on identi.ca or twitter if you know or a format or want to chat about what one would look like.
 digg it   seed it   del.icio.us   ma.gnolia
Comments: 0 Tags:        


Updated Mercurial Config @ 2010-06-03 20:02:01.496252
Filed under: Code  Tech 
I've really taken a liking to using Mercurial for version control. I figured it was about time to post my updated personal config (~/.hgrc)! 

[ui]
username = Steve 'Ashcrow' Milner 
editor = vim

[extensions]
hgext.convert =
hgext.graphlog =
hgext.gpg =
hgext.schemes =
hgext.patchbomb =
color =
mq =
bookmarks =
pager =
inotify =
rebase =

[pager]
pager = less -R

[diff]
git = True

[alias]
blame = annotate -uln

[smtp]
# removed ;-)

 digg it   seed it   del.icio.us   ma.gnolia
Comments: 0 Tags:    


victims demo @ 2010-05-09 00:39:40
Filed under: Code  Tech  Security 

 digg it   seed it   del.icio.us   ma.gnolia
Comments: 0 Tags:      


Bad Java, BAD! No More Jars! @ 2010-03-27 23:26:42
Filed under: Code  Tech  Security  Frustration  Python 
A big frustration for me is the sprawl of Jar's (java "packages") which are everywhere. These special zip files tend to be copied into other applications and then left alone. Many of these Jar's have newer releases to fix security issues, but the bundled version isn't updated. It's even worse that many Jar's don't provide enough meta information that you can be sure of who owns it. Yes, you could keep SHA1SUM's in a database like maven does, and that is better than nothing, but it's not really a fix, it's a hack.

Here is an example of metadata that came with one Jar:

Manifest-Version: 1.0
Created-By: 1.6.0_10 (Sun Microsystems Inc.)

How helpful! Well, we can at least see what it is by the name of the file: sqlitejdbc.jar. It still doesn't tell us what version. Let's look at another:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: fberger
Build-Jdk: 1.6.0_06

Better, but still isn't that helpful. In this case we get lucky as some of the info is in the file name: gettext-commons-0.9.6.jar.

It really seems like the whole 'keep metadata in your Jar' is more of an inside joke which requires a hack to try to track what jars to include (as maven does). Welcome to Jar hell. How fun. Anyway, here is another hack ...

For the heck of it I decided to write a very simple scanner. It reads the metadata from the Jar file and then tries to match it up against an online database. If it gets no results back it keeps it as 'either safe or not enough information'. If there isn't even enough data to make a call out to the database it's assumed bad and tells you the user to bother about it if that is listed, and if the database confirms vulnerabilities it's known bad. It needs a lot of work to better guess information not provided by the metadata but here is an example run (with some changes to protect the guilty) 

$ python jarscanner.py *jar
WARNING:root:apache-mime4j-0.6.jar is the latest secure version or not enough info
WARNING:root:commons-codec-1.3.jar is the latest secure version or not enough info
WARNING:root:commons-logging-1.1.1.jar is the latest secure version or not enough info
WARNING:root:hsqldb.jar is the latest secure version or not enough info
WARNING:root:httpclient-4.0.jar is the latest secure version or not enough info
WARNING:root:httpcore-4.0.1.jar is the latest secure version or not enough info
WARNING:root:httpmime-4.0.jar is the latest secure version or not enough info
INFO:root:jetty-6.1.7.jar found 15 vulns
INFO:root:jetty-util-6.1.7.jar found 15 vulns
WARNING:root:servlet-api-2.5-6.1.7.jar is the latest secure version or not enough info
WARNING:root:The following jars are known to be bad ...
WARNING:root:jetty-6.1.7.jar
WARNING:root:jetty-util-6.1.7.jar
CRITICAL:root:Sorry, but a number of jars are crap and don't provide enough information.
    These should be assumed bad!!!
CRITICAL:root:bdiff.jar
CRITICAL:root:fast-md5.jar (go bug dragonlz about it)
CRITICAL:root:gettext-commons-0.9.6.jar (go bug fberger about it)
CRITICAL:root:jcip-annotations.jar
CRITICAL:root:linuxfolderwatcher.jar
CRITICAL:root:messages.jar
CRITICAL:root:snakeyaml-1.5.jar (go bug somov about it)
CRITICAL:root:sqlitejdbc.jar
CRITICAL:root:stringtree-json-2.0.9.jar
CRITICAL:root:swt.jar
CRITICAL:root:unixapi.jar
CRITICAL:root:XXXXXXXXXX.jar
CRITICAL:root:XXXXXXXXXX.jar


I'm tired. I'm going to bed. I'll throw the code up somewhere tomorrow.
 digg it   seed it   del.icio.us   ma.gnolia
Comments: 0 Tags:          


 
A Django joint.
© 2007-2009 Steve 'Ashcrow' Milner | Studio7designs | Arbutus Photography