Continuous Quality Monitoring

Continuous Quality Monitoring (part 2)

7th June 2015 Continuous Quality Monitoring, Quality management, Software development No comments

In the first post I discussed different kinds of quality and gave some reasons on why the quality tends to suffer as project grows. In this post I’ll try to give some hints on how to keep quality high. If we have our quality goal set then we must use proper tools and actions to enforce the quality. Enforce is a good word here, because quality must be enforced after being measured, because there is a lot of different inputs and changes in today’s projects. Requirements change, developers change, libraries change. We must adapt quickly and provide fast feedback about quality. Sounds like Continuous Integration ? 🙂  Traditional Continuous Integration does include running various tests – unit tests, integration tests mostly, but other tests can also be executed. In case of some projects, for which quality goals include matters like high performance requirements or high security requirements, we must execute performance and security tests on CI server. Not on every build, event not in a build pipeline started by SCM change, but according to a schedule, like nightly builds. If we add this and add some static analysis tools that the process is something more than Continuous Integration, it is Continuous Inspection. Why do this inspections of CI server? Well, I’m fan of automation – Jenkins, Hudson, Bamboo or some other CI server does not get tired (if we don’t include build queues and garbage collection  😉 – but here we can change the CI process or add hardware to run more smoothly), it does not have bad days, does not need coffee and doesn’t fall asleep when reviewing class change number 53.  But let’s get down to the trenches

Here’s what we can monitor (a non-comprehensive list):

  • Code standards – we can use Checkstyle, PMD or some other tool
  • Code metrics – here we can use Checkstyle, Sonar, Dependometer, JDepend
  • Library and classpath issues – JBoss Tattletale
  • Architecture and design issues – Sonar, Dependometer, Structure101, Sonargraph, JDepend
  • Potential bugs – Findbugs, Sonar (including security issues)
  • Bad practices – PMD
  • Potential security vulnerabilities – Sonar, ZED Attack Proxy
  • Insufficient test coverage – JaCoCo (we can break the build if test coverage is not met)

There are more tools (see this link), but I focus here on tools that I have experience with and that can be integrated with Continuous Integration server. Check Google CodePro Analytix – an excellent developer desktop tool to analyze code – plus it it helps to write unit tests, generated comment skeletons. Check tools for architects that can compute quality metrics, visualize code structure and perform simulated refactorings – Dependometer, Sonargraph (a.k.a. SonarJ) and Structure101. Of course JBoss Tattletale also does dependencies visualization but can’t simulate refactoring.

In my opinion Sonar is underestimated tool for a few reasons:

  1. It still grows and new things are being added that go unnoticed (like provisioning and disabling automatic project creation)
  2. It gets misused often – I’ve seen many time Sonar without custom rule sets, that shows like half a million of Dead Local Storage or unnecessary Boxing (BX_BOXING_IMMEDIATELY_UNBOXED) reported on critical (DLS) or major (Boxing) level, that while being important and being a code smell are not critical or even major for 99% projects out there. Here the problem is half a million of one issue report not that the code uses autoboxing and the immediately unboxes.
  3. Often the quality gates feature is not used
  4. Sonar can and will report false positives, so again it must be configured and used appropriately

To use sonar effectively we must configure it according to our projects standards and use quality gates. It should be plugged in into CI process – that is the main point of this discussion, but it can be also used by developers on their desktops. And since using CI server plugin is also more popular I will discuss the usage of Maven plugin that can be used as part of CI build pipeline or on developer desktop.

To maximize the profits from introducing Sonar we want to automate the process then we want the CI server to notify us of error found. We can do this by using Maven and Sonar – if quality of project is below some threshold than the build will be broken by sonar or maven plugin. Static analysis of code with Sonar (including Checkstyle, PMD and Findbugs) is pretty popular so I won’t write about it. But what is maybe not so well known and popular is that we can use Sonar to check some architectural constraints like:

  • Layering
  • Dependency direction

This checks can be implemented in Dependometer or straight in Sonar. Using Dependometer gives more flexibility (like Structure 101), but Sonar provides possibility to break the build as feedback. I like this option, as this forces attention. Sadly there is no Dependometer plugin for Sonar, so we must integrate it with Maven and use build project using Maven on CI server.

In order to analyze project in Sonar we can use CI server plugin or Maven. Below is example fo maven configuration

You can find full list of analysis parameters in Sonar documentation, I don’t want to repeat these docs 🙂 Important thing to note is that we provide Sonar database property as Sonar Maven plugin is an analyzer and stores information in the database so it need to be the same as the one used by Sonar Web app.

We can also use e.g. Jenkins plugin to integrate sonar with Jenkins. After we do this we are able to inspect the code on various levels as Sonar has plugins for Findbugs, Checkstyle, Structure101, JBoss Tattletale (unofficial), PMD and can large rule base of its own (e.g. OWASP TOP 10 violations).

Adding architectural constraint in Sonar is simple – in Rules we search for Architectural constraint and click into the rule


Then we enter name pattern of source and target package specifying that source package can’t depend on target package

In Dependometer we can enter allowed dependencies and specify threshold on various metrics. Specifying allowed dependencies is more convenient in my opinion by both ways give us what we wanted. We monitor no not only project standards, inspect code for bad practices, check for possible bugs but also look for layering violations, cycles, code that can be difficult to extend or maintain (complex, tangled, unstable). Tattletale can check if there are duplicated libraries in our application deployment archive (WAR, EAR – after doing some tricks). Now we have Continuous Inspection 🙂

Hav en dejlig dag !

Continuous Quality Monitoring (part 1)

5th June 2015 Continuous Quality Monitoring, Quality management, Software development No comments

This post will be about monitoring project quality on various levels. This post is an introduction to the subject of continuous quality monitoring. I next will give some technical details how to monitor this quality. In next posts I’ll write about my idea of development process and the place of quality management in it, so if you’re interested check out this blog in a few days. Please check posts given in references at the and of article. They discuss important aspects of quality management that will give you yet another perspective. I would like to focus more on technical details of monitoring.

We’ve seen some projects that had high quality, some with low quality. Most projects start with high quality expectations and promises. Some have standards setup by architect or applied as part of corporate policy. Most project have some sort of automated tests. Sadly many projects lack proper monitoring and continuous quality monitoring. I guess most of us experiences cutting down the effort spend on unit or functional testing. Architecture and quality metrics are measured even less often.

Let’s set common definition of quality from the perspective of quality assurance:

  • internal quality – these is measured on code level. Here we measure quality metrics like distance from main sequence, number of cyclic relations, afferent coupling, efferent coupling, instability etc. We can also check if code is made according to some standards chosen for this project. Business people does not see this quality, but never the less in the end is affected by it.
  • external quality – here we measure if software does what it is meant to do and has proper functional and non-functional characteristics. Business people see this type of quality and this gets more attention. This type of quality is about functional testing, running various security checks, performance, load and stress testing the application. We can measure some architecture level quality metrics like security, reliability, performance.

For majority of projects we cannot completely separate the two because they have the same source. If code has poor quality then probably it will not be scalable, secure or reliable. It will have a lot of bugs and will be difficult to maintain. And most projects start with high expectations and good will. So what goes wrong? It is hard, if possible, to give definitive answer but from my experience I can pinpoint a few reasons:

  • Lack of common quality vision and commitment to this vision – management, business and engineers must have a common understanding of quality, consequences of low quality, and level of quality they want to achieve. Showing examples of failed projects here will help to get a common understanding. On the other hand engineers have to remember that software is not an art work made for the sake of making software.
  • Lack of continuous inspection – continuous integration with continuous inspection is a must in todays projects. We simply don’t have time and money to discover failures later in the process of making software.
  • Lack of proper quality assurance – this is an interesting point. A common example is writing tests to achieve x% coverage, which is simply nothing more than throwing away money. Show me business that wants to throw away money 😉 Another example, mentioned by Petri are incorrectly introduced code reviews.
  • Lack of proper architecture and monitoring of implementation of these architecture – requirements will change, especially if the product is successful, so it must be prepared to be extended and modified without harm to its quality.

What can we do ? We can focus on our quality goal and measure it. I won’t discuss how we can setup quality goal, as I believe this depends on organization, its standards and culture. I want to focus on technical aspects of measurements.

References to articles: