Let me be clear about this: This isn’t an open source issue, and we shouldn’t level blame on open source users and producers (Full disclosure: my company Sonatype is an open source software development firm).  Economic and production efficiencies of open source have made it an almost compulsory component of any modern software application. We’ve all reaped tremendous benefits from open source – we develop fast, re-use proven components, and can focus more time on the functionality that’s truly valuable to our employers.

It’s not just that open source is good – it’s necessary. That’s why more than 70,000 organizations made nearly 8 billion requests for open source components from the Central Repository last year for use in all the major categories of applications, including the web, cloud, mobile and critical infrastructure.

The hard truth is that today more than 80 percent of a typical software application is assembled from existing components – and the vast majority of those are open source, coming from dozens, if not hundreds, of individual projects. All industry verticals, both regulated and unregulated, are using tremendous amounts of open source components in both internal and consumer-facing applications.

Open source is essential

Think of software development organizations today the same way you would think of car manufacturers. Developers assemble applications using existing components or parts rather than writing applications from scratch. But unlike manufacturing, the software industry lacks the tools to manage the intricacy and risk associated with a complex and distributed software supply chain.

Component-based development needs to be managed, for sure; security problems arise when oversight is incomplete. Simply put, a flawed software supply chain means flawed applications. Our research indicates that at least 71 percent of applications contain components with known security flaws that are classified as severe or critical.

According to one study, “The Leaking Vault 2011” by the Digital Forensics Association, more than $156 billion in direct losses can be attributed to data breaches in just a five-year period. The Application Risk Management in Business Survey by Forrester and Veracode found that 62 percent of surveyed organizations reported breaches in the past year due to flaws in their critical applications.

Mitigating inevitable risks

The question becomes then how to mitigate the risks associated with component consumption while realizing the benefits of open source. Certainly there are constant and sophisticated threats to open source software; this is true with proprietary software too. We know where danger lies: it comes from using outdated components with known vulnerabilities. It comes from not having an enforceable open source policy. And it comes from not managing component licenses or the licenses of dependencies.

It is important to understand that this is a supply chain problem: You need to manage components at each phase of the software development lifecycle — at consumption, in development, during integration and within production.

Decreasing security exposure

Decisive security measures at the component layer strengthen the entire software development lifecycle and increase the integrity of the overall software supply chain. Imagine the risk of a vulnerability in a popular open source component. Because the component is used in many applications, within and across various organizations, it becomes a rich target for exploitation. History has shown that an attacker only needs to gain a foothold in an organization and often attacks the weakest link, so the risk of component-based software development could not be greater.

Here are the keys to decreasing this type of exposure:

• Institute an open source policy if your organization doesn’t already have one. If you do have one, review it, and often.  Make sure it’s clear to both those on the development team and those responsible for managing the security process – whether it’s risk management, legal or Jim the senior developer – to get buy-in from everyone.
• Ensure your policy includes key guidelines for component security, licensing and quality attributes. Beyond the over-arching open source policy that outlines the organization’s standards and values, create additional guidelines to drive usage decisions (e.g. age of component at download, license-type, level of documentation.)
• Be sure your policies are enforceable. Without the ability to enforce, honestly, what’s the point?  Paper-based policies will be ignored, so look for ways to integrate enforcement into the software development process itself.
• Give developers the information they need to make good choices. Your developers are on the front lines so give them the ability to fight. Detecting flaws or non-compliance early on in the process saves time and money down the line.
• Before going to production, inventory components and their dependencies. Knowing the makeup of your application is half the battle in troubleshooting vulnerabilities that may be discovered later.
• Continuously monitor for newly discovered flaws. New vulnerabilities emerge all the time (just like in proprietary software). You need to know when a new flaw is surfaced, and where exactly the component is being used.
• Have a remediation plan. Know how you can fix problems regardless of where they occur in the lifecycle.  Fixing flaws isn’t always easy, and having a plan helps.

Open source or proprietary, free or paid, remember this: If we can safeguard the component layer by instituting good component practice it will pays dividends across the lifecycle.

Ryan Berg is Chief Security Officer of Sonatype and former Cloud security strategy lead at IBM

Have an idea for a post you’d like to contribute to GigaOm? Click here for our guidelines and contact info.

Photo courtesy pryzmat/Shutterstock.com.

Follow the simple steps below to get one of the best repository managers on the market up and running in a few minutes.

Create the environment

1. Go to jelastic.com and sign up if you haven’t done it yet or log in with your Jelastic credentials by clicking the Sign In link on the page.

2. Ask Jelastic to create a new environment.

3. Choose Tomcat 7 as your application server and set the cloudlets limit. Then type your environment name, for example, nexus, and click Create.

Wait just a minute for your environment to be created.

Upload Java package

1. Go to Sonatype web site and copy the link to the latest Nexus release WAR archive.

2. Switch back to the Jelastic dashboard and upload the java package to the Deployment manager using copied URL.

3. Once the package is in Jelastic, deploy it to the environment you have created earlier.

4. Open your application in a web browser and start managing your artifacts!

Cheers!

Nearly 80% of the enterprises surveyed consume open source software. Most interesting to me: two thirds of them are actively contributing code back to the upstream projects they consume. Also interesting to note is that just shy of half of all surveyed companies have a formal open source policy in place. And of those with formal policies, half of the respondents cite those policies as detrimental to the success of development.

The top complaints about formalized open source policies are:

• it slows down development
• we find out about problems too late in the process
• it’s not clear what’s expected of us
• there is no enforcement

Some organizations further restrict open source software usage by license, going so far as to verify the license of all components and their dependencies. At first blush that might sound like a big waste of time, but in reality that’s a good thing: open source license compliance is important, and fundamentally important to the longevity of open source in general. Of course, if these enterprises aren’t distributing their applications to others then license compliance with copyleft licenses like the GNU Public License isn’t as big of a deal.

Sonatype’s primary product, Nexus Professional is a repository manager that aims to solve many of the licensing, dependency, and procurement problems identified in the Sonatype survey (again, with a specific focus on Java). The survey highlights that 73% of enterprises stay informed on new releases of the open source components they use by manual web searches, or by directly visiting the projects’ websites. That’s clearly inefficient. Even established code sharing services like GitHub, Google Code, and SourceForge aren’t being as heavily utilized as they could be by the companies surveyed.

The primary motivation for this report is to demonstrate the need for Sonatype’s products, obviously. That focus, though, reveals useful general information. For example, the financial industry is the most likely to completely lock down open source developers to using specific approved resources. Another interesting revelation: the aspects of open source software most important to the companies surveyed are maturity, security, and overall code quality. License type is only of interest to a comparatively smaller portion of survey respondents.

Regardless of any challenges introduced by open source software, it’s clear that open source is gainin more popularity within traditional enterprises. Anything that can be done to simplify the consumption and compliance issues identified by Sonatype — for Java and every other language — is a good thing.

For businesses and other organizations today, open-source software (OSS) is transformative in terms of its ability to allow organizations to write software very quickly and to leverage innovation very aggressively.

OSS component-based development has reached a strategic tipping point, having moved from a cost-effective solution to a competitive advantage capable of delivering rapid and substantial return on investment for organizations that use it.

And use it they do. More than 80 percent of modern software includes open sourcecomponents.

Typical organizations, including Global 2000 enterprises, use thousands of OSS components, often in mission-critical software portfolios. Startups can quickly bring applications to market by focusing creative development on their core competency and relying on OSS for everything else. My company’s Central Repository, containing nearly 90 percent of open source Java projects, serves more than four billion requests per year to more than 61,000 organizations per year, including more than half of the Global 2000.

A vibrant ecosystem with a fundamental flaw

The same core advantage of OSS –its free availability, rapid innovation, and highly interdependent projects — introduces risks that can sabotage the IT or business value of key applications.

The issue really boils down to two interrelated concerns:

Complex dependencies: The open source ecosystem is comprised of hundreds of thousands of components, each of which may depend on tens or hundreds of other components. The whole ecosystem is interdependent. As a result, the properties (good or bad) of any one component are inherited across many others.

A simple, but potent example might help. Version 2.5.6 of the Spring-beans contained a severe, remotely exploitable security flaw. Spring-beans is a commonly used component, and 1,447 others depend on it. So the security vulnerability was inherited by all 1,447 other components, and untold thousands of applications that rely Spring-beans directly or indirectly.

Intellectual property issues add another dimension. Every component and dependency added to an application has specific and enforceable licensing and copyright requirements. This is true even if those dependencies are added unwittingly. This is troubling for software and embedded systems vendors who might inadvertently include a copyleft license such as the GPL in their shipping products.

This exact issue has resulted in numerous and expensive lawsuits including the well-publicized instance of Cisco’s unknowing inclusion of GPL code in their Linksys routers. In this case, the Free Software Foundation sued Cisco and forced the company, among other things, to make their source code publicly available.

Lack of update notification infrastructure: Components are updated frequently; the average component in Central is updated four times year. And yet, with all this change, there is no automated mechanism for update notification.

Take the Spring-beans example. Once the security vulnerability was fixed, there was no automated mechanism for the projects that depend on the old version to be updated to the new, fixed version. Taking it one step further, absent automated update notification, none of the direct or indirect users of the flawed Spring-beans components would have any idea that their applications were at risk.

In this wild West sort of lawlessness, many organizations are clearly taking chances and hoping for the best. A 2012 survey we conducted among 2,550 developers, architects, and managers found that only 20 percent of organizations have put effective open source management policies in place.

Order out of chaos: A strategy for optimization

Strategizing to yield the greatest ROI in using OSS demands a high-level awareness of how, why, and where OSS is used, along with consistent knowledge of OSS benefits, risks, and policies.

To this end, several vendors offer software composition analysis tools that apply data mining technology for use in inspecting OSS components for security and functionality issues, known fixes, IP ownership, and versioning. The best of these tools enable organizations to govern development processes, continuously monitor the health of their repositories, and retrieve real-time alerts when critical applications are affected by newly discovered threats.

To maximize the business value of OSS while minimizing risks:

• Assess your current usage of OSS components to grasp where you’re starting from, as an aid to setting realistic goals.
• Establish an open source governance program to filter, audit, track and manage open-source assets in the organization, and deploy mechanisms to monitor the effectiveness of your governance program.
• Build open source management into your entire software development process, evaluating OSS components before and while using them in development .
• Analyze and continuously monitor all deployed applications for newly discovered security vulnerabilities and stability issues.
• Establish well-defined channels of acquisition (such as the Central Repository) for each OSS component you leverage.
• Engage with the OSS community and establish routes to service and support for key components and frameworks.

Properly managing the use of OSS in development will let you focus not merely on the cost savings it can bring you, but also on the wealth of innovation ongoing in the open source domain. It will help make OSS a catalyst for change in your organization.

Wayne Jackson is CEO of Sonatype, a company that is transforming software development with tools, information, and services that enable organizations to build better software faster using open-source components. Contact him at wjackson@sonatype.com.

Image courtesy of olly, Shutterstock

The project that I published is the FFPOJO Project, which is a Flat-File Parser, POJO based, library for Java applications.

The objective of this post is to comment my experience in the publishing process and provide some tips and points to people who are in this process to.

• It’s quite obvious but you must own the domain in your GroupId and the package names must start with this domain too;
• If you host your project at GitHub, you can create a free account and turn it on an organizational account, then link this account to your personal user as an organization member. This can provide you an domain like “https://github.com/projectname”, which let you to use the group id and package names like “com.github.projectname”;
• Create a parent-pom and use inheritance and multi-modules to concentrate project-specific pom tags into the parent-pom and facilitate the release process. Remember that the nested project structure is more compatible than flat structure;
• Follow strictly the instructions in the official Sonatype repository usage guide;
• Use the javadoc pluging and the source plugin to generate the “-javadoc.jar” and “-sources.jar”;
• Use the maven release plugin to facilitate the release management;
• If you use GitHub and Windows for development, you might get an error on release:prepare that maven stucks after the push command. This happens when you use passphrase in your GitHub SSH certificate. The best solution I found is to use other certificate with no passphrase. I found other solutions like use putty pageant/plink to cache the certificate and passphrase and use it as ssh client, but not worked for me. The easiest is not use passphrase at all;
• If you use GitHub and Windows for development, when you call release:prepare on Cygwin Git Bash, you might see an error like “pom.xml is outside repository”. It’s a relative/absolute path trouble in the maven git scm plugin. The best solution I found is to run maven from cmd.exe instead of Git Bash. Then the ssh.exe and git.exe folders must be in the PATH variable;
• Don’t forget to publish your GPG public key at the public keyservers (hkp://pool.sks-keyservers.net, hkp://keyserver.ubuntu.com, hkp://pgp.mit.edu). This is verified by the central at release promotion time.

That’s it. I have not found any other trouble in the process, it is very fast and the JIRA administrators are very attentive. Finally, the FFPOJO framework can be found in the Maven Central like any other open source framework:

<dependencies>
	<dependency>
		<groupId>com.github.ffpojo</groupId>
		<artifactId>ffpojo</artifactId>
		<version>1.0</version>
	</dependency>
</dependencies> 

If you want to know better the FFPOJO library, please visit the project website on GitHub.

First of all it’s important to understand that RTC v3 has limited support for Maven. You can setup a build definition which can call a maven goal (such as a local install) easily enough but that’s about it. The RTC build mechanism is basically a cron job with a nice web (like Hudson/Jenkins without the plugins) and Eclipse front end, it doesn’t have a deep understanding of Maven.

The whole point of Maven over Ant is that it does a lot by convention rather than specification, however in most advanced Maven project I’ve seen pom files are as big as equivalent Ant + Ivy files. Regardless I still think Maven with the excellent Sonatype Nexus is awesome (although I also like the structured bash/batch like nature of Ant as well).

Most serious Maven projects are multi-module. This is because component based development is a good thing (shameless employer plug, but it’s true, it is a good thing). Most (tending to all) multi-module projects have the following structure where “O” is a folder and “-” is a file, considering a hypothetical ProductX:

- pom.xml (the parent pom for the master/uber/product build, lists child modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff (such as src)

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

If you want to work with this stuff in RTC Jazz SCM then you need to balance the constraints of Maven m2, RTC and Eclipse. If you’re considering doing this then you’ll probably be considering moving existing Maven projects to RTC, in which case you may find a previous blog on a manual migration script useful. The only edit to this I make for Maven is to separately import the parent from the children if flattening the structure (although in most cases I wouldn’t recommend flattening).

The constraints playing against each other are:

• Maven and plugins expect sub-modules to be in child directories as in the tree above
• RTC doesn’t allow both an Eclipse project and root files to be at the root of a component
• The simple case of Eclipse and RTC SCM interacting during a load from the server restricts Eclipse to only seeing projects which are represented as folders at the root of the local Eclipse workspace containing .project files.
• Eclipse doesn’t really support parent and child projects.

I’ve found there are a number of approaches, none are perfect:

1. Flatten the structure (my opinion the worst option)

You can edit the parent pom.xml to reference the sub-modules as ../ModuleA etc. although the Maven release plugin (amongst others) don’t play well with this. Also, and somewhat fundamentally, you shouldn’t have to restructure your code based on your SCM tool.

2. Only load the relevant bits (my opinion is it’s sometimes ok)

You could just load the child modules you’re currently working on and then unload (remember to delete local files or you hit a bug) and then reload the parent when you want to do a “big” parent build. Frankly this is unacceptable from a developers perspective and will fail anyway if there’s a dependency on the parent from the children. Although in theory this may be frowned upon it’s quite common and so needs supporting by the scm tool.

3. Load then import, with auto-refresh (my preferred option)

This one is going to be more than one paragraph….

Prediction fulfilled ;p Anyway… Currently your developers may be used to a scm system where they load/copy/access files from the scm system on the local filesystem and then do an Eclipse -> Import Existing Maven projects to get all of the various goodies.

If you import the multi-module project as above but with a root folder (as an eclipse project) so it has the following structure:

O Product X

.project

- pom.xml modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

Then when you do a load from Jazz SCM you’ll get in the Eclipse Project Explorer (or equivalent) ProductX correctly detected as a Maven project (or an Eclipse project with a Maven nature for the pedantic amongst you).

You can then do a normal Eclipse -> Import Existing Maven Projects based on your workspace/ProductX/… (taking care not to select “Copy projects into workspace”) You’ll then get in Project Explorer (or whatever):

ProductX (info about rtc scm links)

ModuleA

ModuleB

In this view ModuleA and ModuleB exist in two places: as projects referenced in the workspace but apparently outside of RTC SCM control; as a folder structure under ProductX/ModuleA and ProductX/ModuleB.

This has the following advantages:

• Cross-project dependencies resolve
• You can run the parent/uber pom goals and the child pom goals without reloading anything
• You can shift focus between the child-modules and parent structure without reloading anything

If you edit files in ModuleA and ModuleB under the folder hierarchy of ProductX then Pending Changes will keep track in the normal way but if you edit a file under the apparent “root level” ModuleA and ModuleB then Pending Changes won’t unless you do a “deep refresh”.

Alternatively just go to Eclipse -> Window -> Preferences ->  General -> Workspaces -> Refresh automatically (er… I think, I’m doing this from memory, hence the lack of screenshots). You can then add “edit files tracked by Pending Changes at either Product or child module level” to the advantages above.

So this option sounds great so far, what are the downsides?

This solution maintains the hierarchical structure between ProductX and it’s child modules and as such constrains everything from ProductX downwards to be in the same Jazz SCM component as eclipse projects can only be seen (without far too much fiddling) if they’re root folders in components. This may not seem like such a big deal but components are the lowest level of baseline (or label) in Jazz SCM. In a tightly coupled cohesive single architecture this might not be a problem but if you want to reuse any of these modules (at a code level) or have a separate ownership it will be a problem. It’s also very common to have dependencies from child modules to parent projects (which may initially seem to resolve if they’re in the local ~/.m2 repository).

Having said that source code dependencies between components lead to brittle architectures, binary dependencies are far superior, and made much easier by the use of things like Ivy/Maven and Nexus.

Other considerations

The point above above binary vs. code dependencies can be argued until the cows come home and go out again (an extended metaphor), especially if you consider project/product/team/release boundaries. I’m not going to pretend to answer those with an academic silver bullet here, the important thing I hope you take away from this blog is that there are pros and cons to all of these approaches, balancing those with the needs of your team/architecture/project/organisation are not easy. Shameless plug: Give me a call if you need help

Eclipse refactoring isn’t dealt with well by Jazz SCM, especially between Eclipse projects in and “out” of source control (advocated by option 3 above). Expect a lot of “remove/add” combinations if you are doing anything more than a trivial set of file changes. This is also a problem for the ClearCase <-> RTC automatic synchroniser developed by the excellent Samecs however I know that I, others and Samecs are pushing for improvements on this from IBM.

This post has been very scm focussed but what about build and release? The RTC build engine understands maven to a limited degree and can easily invoke maven builds for information on integating the maven release process see How to do maven releases with Jazz SCM

Image via CrunchBase

Recently, when I decided to make my projects available to the Java Community under Open Source license, I did a lot of things like Hosting my code on Google Code, using Maven to build and test the code and also making sure that I have a project site up and running. However, even after all of this, there was one thing that was missing which was the most essential – “Hosting the project files on a repository so that users who may find these utilities useful may use it”.

I found out that Sonatype would allow me to deploy my code to Maven Central Repository. However, after 3 days of trial and error, I have finally been able to release a version of my code to the Central Repo. I am writing this post so that if you ever want to do this, you will not have to go through the same pain as I did.

Read the full article here (http://scratchpad101.com/2011/09/08/project-files-maven-central/).

Ansonsten beschreibt das Vimeo auf dieser Seite die Installation sehr gut. Und das downloaden der Artefakte beschleunigt sich erheblich!

Noch ein kurzer Nachtrag:
Nach ein Paar Tagen ging’s mir ziemlich auf den Keck’s den Nexus immer von Hand starten zu müssen. Daher hier noch der Tipp wie man den Nexus unter OS X auf dem Mac beim einloggen hochfahren kann.

Als erstes brauchen wir hierzu eine .plist Datei

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
	<dict>
		<key>Label</key>
		<string>sonatype.nexus</string>
		<key>Program</key>
		<!-- Enter the path to your nexus installation here -->
		<string>/usr/local/nexus/nexus-current/bin/jsw/macosx-universal-32/nexus</string>
		<key>ProgramArguments</key>
		<array>
			<string>nexus</string>
			<string>start</string>
		</array>
		<key>RunAtLoad</key>
		<true/>
	</dict>
</plist>

Diese speichern wir unter dem Verzeichnis
~/Library/LauchAgents/sonatype.nexus.plist
Als Nächstes tragen wir das Ganze mit dem launchctl Kommando auf der Console ein
launchctl start sonatype.nexus

Das war’s dann auch schon. Ab jetzt sollte nach dem einloggen des Users im Browser unter localhost:8081/nexus die Adminconsole von Nexus erscheinen.

Die Idee hab ich übrigens von hier. Aber die .plist Datei von dort funktioniert absolut nicht – sieht auch sehr experimentell aus. Würd’ dem Autor auch gern in seinem Blog schreiben, aber extra dafür anmelden??? :s

I’ll be regularly making an ass of myself for “Sonatype Week in Review” over the coming months, if you would like to subscribe to the Podcast feed in iTunes click here.