Bash-fu part2: highlight important stuff in logs and Maven build

Some time ago when I was staring at the log files, not much fun by the way, I said to myself that it would be nice to have a way to highlight information that is more important than the rest. I know there is plethora of log analysis tools that allows¬†you to filter on the log data, but I don’t want tool with GUI, I like terminal.

With using sed and terminal colors it is a piece of cake. Let me post here a snippet of my ~/.bashrc file and I’ll comment it later on.

##### </Maven colors>
# Mvn color (https://gist.github.com/1027800) 
# Formatting constants
export BOLD=$(tput bold)
export UNDERLINE_ON=$(tput smul)
export UNDERLINE_OFF=$(tput rmul)
export TEXT_BLACK=$(tput setaf 0)
export TEXT_RED=$(tput setaf 1)
export TEXT_GREEN=$(tput setaf 2)
export TEXT_YELLOW=$(tput setaf 3)
export TEXT_BLUE=$(tput setaf 4)
export TEXT_MAGENTA=$(tput setaf 5)
export TEXT_CYAN=$(tput setaf 6)
export TEXT_WHITE=$(tput setaf 7)
export TEXT_ORANGE=$(tput setaf 172)
export BACKGROUND_BLACK=$(tput setab 0)
export BACKGROUND_RED=$(tput setab 1)
export BACKGROUND_GREEN=$(tput setab 2)
export BACKGROUND_YELLOW=$(tput setab 3)
export BACKGROUND_BLUE=$(tput setab 4)
export BACKGROUND_MAGENTA=$(tput setab 5)
export BACKGROUND_CYAN=$(tput setab 6)
export BACKGROUND_WHITE=$(tput setab 7)
export RESET_FORMATTING=$(tput sgr0)
 
# Wrapper function for Maven's mvn command.
mvnColor() {
  mvn $@ | sed -e "s/\(\[INFO\]\ \[.*\)/${RESET_FORMATTING}\1${RESET_FORMATTING}/g" \
               -e "/\[INFO\]\ Building\ \(war\:\|jar\:\|ear\:\)/! s/\(\[INFO\]\ Building .*\)/${TEXT_BLUE}\1${RESET_FORMATTING}/g" \
               -e "s/\(\[INFO\]\ BUILD SUCCESSFUL\)/${TEXT_GREEN}\1${RESET_FORMATTING}/g" \
               -e "s/\(\[INFO\]\ BUILD FAILURE\)/${TEXT_RED}\1${RESET_FORMATTING}/g" \
               -e "s/\(\[WARNING\].*\)/${TEXT_YELLOW}\1${RESET_FORMATTING}/g" \
               -e "s/\(\[ERROR\].*\)/${TEXT_RED}\1${RESET_FORMATTING}/g" \
               -e "s/\(Caused by: .*\)/${BOLD}${TEXT_ORANGE}\1${RESET_FORMATTING}/g" \
               -e "s/\(org\.rhq\..*\)/${BOLD}\1${RESET_FORMATTING}/g" \
               -e "s/Tests run: \([^,]*\), Failures: \([^,]*\), Errors: \([^,]*\), Skipped: \([^,]*\)/${BOLD}${TEXT_GREEN}Tests run: \1${RESET_FORMATTING}, Failures: ${BOLD}${TEXT_RED}\2${RESET_FORMATTING}, Errors: ${BOLD}${TEXT_RED}\3${RESET_FORMATTING}, Skipped: ${BOLD}${TEXT_YELLOW}\4${RESET_FORMATTING}/g"
  echo -ne ${RESET_FORMATTING}
}
alias mvn="mvnColor"
##### </Maven colors>


logColor() {
  echo $@
  [ $# = 0 ] && exit
  $@ | sed -e "s/\(\ INFO\ \ .*\)/${RESET_FORMATTING}\1${RESET_FORMATTING}/g" \
           -e "s/\(\ WARN\ \ .*\)/${TEXT_YELLOW}\1${RESET_FORMATTING}/g" \
           -e "s/\(\ ERROR\ .*\)/${TEXT_RED}\1${RESET_FORMATTING}/g" \
           -e "s/\(org\.rhq\..*\)/${BOLD}\1${RESET_FORMATTING}/g" \
           -e "s/\(Caused by: .*\)/${BOLD}${TEXT_RED}\1${RESET_FORMATTING}/g"
}

alias serverLog="logColor tail -f $RHQ_HOME/dev-container/rhq-server/logs/server.log"

There are two functions that have a lot of in common and it certainly could be refactored not to repeat the common parts, but let’s leave it aside.

I think the code is pretty self explaining, but for sure. What it does is that it kind of sniffs on the output of the watched command (mvn or tailf) and it tries to replace some patterns with the same content but surrounded by color tags. So for instance if in the output of Maven build there is a line containing “Building jar”, it will be blue. Test failures, build failures, errors and causes of exceptions are red. Warnings yellow, successful tests green, etc. I also use a bold highlight for any lines with “org.rhq.” pattern. This is here, because in the long exception stack trace I want to recognize where it talks about the code I own (no 3rd parties noise).

Package highlighting FTW

Package highlighting FTW

The same principles apply for the second function for log highlighting. It has one small drawback, if I redirect the output to a file, there is a mess there because of the colors, but it is easily solvable.

Donate

I hope, it’ll help you to find problems more quickly. If you find this information helpful and want to support me, you can send some small coin to my BTC wallet 1MKHbfwSYrYs2gZpiRupoJhaEkuXq6XNLL.

puppy dog eyes

bitcoin-nervy

Posted in Java, Tools

Improve your Bash-fu with Command-line Completion

I am a Java developer and I am used to certain level of comfort when using all the tools Java ecosystem provides. All the modern IDEs contains code completion feature for example. I use Bash a lot and I use it mostly for git manipulation, because I find all the IDE plugins kind of lame. Fortunately, all the git packages provide also the bash completion, so if I can’t recall a certain option/switch, instead of “RTFM”, I can type in just:

[jkremser@jk ~ ]$ git grep --[TAB]

and I get

[jkremser@jk ~ ]$ git grep --
--all-match       --count
--and             --extended-regexp
--basic-regexp    --files-with-matches
--cached          --files-without-match
--fixed-strings   --line-number       --or
--full-name       --max-depth         --perl-regexp
--ignore-case     --name-only         --text
--invert-match    --not               --word-regexp

That certainly saves some time. If you were wondering how this is done or want to implement it for your tool, keep reading.

Bash Completion in Detail

Actually, it is pretty easy. It requires to install (or ensure it is installed) the package called bash-completion. Now, let’s stick with the git example. If you type in:

[jkremser@jk ~ ]$ declare -F

It’ll write all the functions defined in the current shell. If the git is installed, you should be able to find functions starting with _git prefix. Function responsible for completing the "git grep --[TAB]" (as shown above) is called _git_grep, so you can see, how it looks like by:

declare -f _git_grep
_git_grep ()
{
    __git_has_doubledash && return;
    case "$cur" in
        --*)
            __gitcomp "
			--cached
			--text --ignore-case --word-regexp --invert-match
			--full-name --line-number
			--extended-regexp --basic-regexp --fixed-strings
			--perl-regexp
			--files-with-matches --name-only
			--files-without-match
			--max-depth
			--count
			--and --or --not --all-match
			";
            return
        ;;
    esac;
    case "$cword,$prev" in
        2,* | *,-*)
            if test -r tags; then
                __gitcomp_nl "$(__git_match_ctag "$cur" tags)";
                return;
            fi
        ;;
    esac;
    __gitcomp_nl "$(__git_refs)"
}

So far no rocket science, right? I am not going to go into detail with the git completion, because it uses a lot of helper functions that are good for making the code DRY, but not so much for demonstrating the concepts. I’ll show how one can write it’s own bash function for the bash completion. Say, you’ve written a CLI tool in your favorite language and you would like to let the terminal to whisper you the right options on the right places.

Let’s assume the program is written in Java and it is packed as a jar file and the valid options are foo, bar and baz.

public class Cli {
  public static void main(String... args) {
    if (args.length != 1) {
      System.err.println("invalid usage: number of params");
    } else if ("foo".equals(args[0])) {
      System.out.println("Hello foo");
    } else if ("bar".equals(args[0])) {
      System.out.println("Hello bar");
    } else if ("baz".equals(args[0])) {
      System.out.println("Hello baz");
    } else {
      System.err.println("invalid usage: unknown param " + args[0]);
    }
  }
}

Create the jar file for this simple example: (or use IDE magic)

$ javac Cli.java && jar cfev cli.jar Cli Cli.class

Now, instead of calling it by

$ java -jar cli.jar {foo|bar|baz}

Let’s create a bash script and place it on $PATH. Make sure you replace the path to the cli with the right path.

$ su --
$ echo -e '#!/bin/bash\njava -jar /home/jkremser/blog/cli.jar $@' > /usr/bin/cli
$ chmod +x /usr/bin/cli

Allright, now we can call just cli from anywhere, but the completion still doesn’t work, let’s fix it.

Create following bash script, call it cli-completion.sh, mark it as executable and place it to the /etc/bash_completion.d directory.

#!/bin/bash

_cli() {
    COMPREPLY=()
    local cur _opts
    _cur="${COMP_WORDS[COMP_CWORD]}"
    _opts="foo bar baz"
    COMPREPLY=( $(compgen -W "${_opts}" -- ${_cur}) )
}

complete -F _cli cli

Now, if you run a new shell you should be able to get the suggestions when hitting TAB key.

[jkremser@jk ~ ]$ cli [TAB]
bar  baz  foo

Bash Completion for RHQ

Using the described steps, I’ve created a bash completion script for rhqctl that serves as an command line based entry point to RHQ. Instead of creating a bash script it’s only purpose was to redirect the call to java -jar something + passing the arguments, I’ve used simple bash alias feature, both options are viable. All the allowed options and switchs are described on our wikipage. Here is the code completion script:

#!/bin/bash

# completion function for rhqctl command
# to auto load this script move this script to /etc/bash_completion.d/

_rhqctl() {
    local cur prev opts agentServerStorage agentServerStorageStart serverStorage dataMingratorSubopts \
          storageInstallSubopts agentInstallSubopts serverInstallSubopts upgradeSubopts booleanSubopts trueFalse first second
    COMPREPLY=()

    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts="console install restart start status stop upgrade --help"
    agentServerStorage="--agent --server --storage"
    agentServerStorageStart="--agent --server --storage --start"
    serverStorage="--server --storage"
    dataMigratorSubopts="none estimate print-command do-it"

    # the spaces in the beginning and in the end are important here
    storageInstallSubopts=" --storage-data-root-dir "
    agentInstallSubopts=" --agent-config --agent-preference "
    #serverInstallSubopts=" --server-config "
    upgradeSubopts=" --from-agent-dir --from-server-dir --start --run-data-migrator --storage-config --storage-data-root-dir --use-remote-storage-node "
    booleanSubopts=" --use-remote-storage-node "
    trueFalse="true false"

    if [[ "${COMP_LINE}" =~ ^\.?rhqctl[[:space:]]*$ ]] ; then
      COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
      return 0
    fi

    first="${COMP_WORDS[1]}"

    if [[ "x${first}" == "xstart" ]] || [[ "x${first}" == "xstop" ]] || [[ "x${first}" == "xstatus" ]] || [[ "x${first}" == "xrestart" ]]; then
        COMPREPLY=( $(compgen -W "${agentServerStorage}" -- ${cur}) )
        return 0
    elif  [[ "x${first}" == "xconsole" ]] ; then
        COMPREPLY=( $(compgen -W "${serverStorage}" -- ${cur}) )
        return 0
    elif [[ "x${first}" == "xupgrade" ]] ; then
        if [[ "${upgradeSubopts}" == *" ${prev} "* ]] ; then
          if [[ "x${prev}" == "x--run-data-migrator" ]] ; then
            COMPREPLY=( $(compgen -W "${dataMigratorSubopts}" -- ${cur}) )
            return 0
          elif [[ "x${prev}" == "x--start" ]] ; then
            COMPREPLY=( $(compgen -W "${upgradeSubopts}" -- ${cur}) )
            return 0
          else
            checkForBooleanSubopt ${prev} && return 0
            completePath ${cur}
            return 0
          fi
        fi
        COMPREPLY=( $(compgen -W "${upgradeSubopts}" -- ${cur}) )
        return 0
    elif [[ "x${first}" == "xinstall" ]] ; then
        second="${COMP_WORDS[2]}"
        if [[ "x$second" == "x" ]] || [[ "x${second}" == "x--" ]]; then
            COMPREPLY=( $(compgen -W "${agentServerStorageStart}" -- ${cur}) )
            return 0
        fi
        if [[ "x${second}" == "x--agent" ]] ; then
            if [[ "${agentInstallSubopts}" == *" ${prev} "* ]] ; then
                completePath ${cur}
                return 0
            fi
            COMPREPLY=( $(compgen -W "${agentInstallSubopts} --start" -- ${cur}) )
            return 0
        elif [[ "x${second}" == "x--storage" ]] ; then
            if [[ "${storageInstallSubopts}" == *" ${prev} "* ]] ; then
                completePath ${cur}
                return 0
            fi
            COMPREPLY=( $(compgen -W "${storageInstallSubopts} --start" -- ${cur}) )
            return 0
        fi
        COMPREPLY=( $(compgen -W "${agentServerStorageStart}" -- ${cur}) )
        return 0
    # fallback
    elif [[ "${cur}" == * ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        return 0
    fi
}

function completePath(){
  compopt -o nospace
  COMPREPLY=( $(compgen -f "${cur}") )
}

function checkForBooleanSubopt(){
  if [[ "${booleanSubopts}" == *" ${prev} "* ]] ; then
    COMPREPLY=( $(compgen -W "${trueFalse}" -- ${cur} ) )
    return 0
  else
    return 1
  fi
}

complete -F _rhqctl rhqctl

That is it!

That is it

I hope you find this blog post helpful. I am not a Bash ninja so bare with my bad habits. I would appreciate any suggestions for improvements in the comments. Hopefuly, it won’t be my last post for next 2 yearsūüėČ

Posted in RHQ, Tools

Java Developer Day 2012

On my way back home from the Red Hat new hire orientation in Munich I attended the developer conference in Prague called Java Developer Day 2012. Last year it was held in Brno and, I must say, it was much better than this year. Not only the all the topics were nearly the same, but the place was definitely worse this year. To me it looked like some storage building, c’mon Oracle. Instead of saying “we will introduce feature X in a couple of months” as they did last year, they were now saying “feature X is almost done”, however, in general, the topics were the same.

The talks were separated into two tracks: Java SE and Java EE. There were also two versions of the keynote (SE and EE). The conference started with the Java SE keynote where¬†Alexander Belokrylov, the Java evangelist, was trying to resurrect the Java FX. Not only to me, it is the dead technology and noone cares about the desktop nowadays. I must admit the demo app was pretty cool, he had a Kinect plugged together with the demo app and as he was moving, the Duke was doing the same. More than demo for Java FX it was demo for Open GL, though. The only interesting idea he presented (imho) is using the Java FX for building iOS apps using embedded JRE (each app will have it’s own JRE). On the other hand, I don’t think that Apple will like this idea and it will be slower than the native Objective C apps.

The topic of the second keynote was Java EE, it was presented by Anil Gaur, the Vice President of Oracle for Java EE. Mostly, he was talking about future development, about the need for multitenancy in a cloud environment, about better support of PaaS solutions, even about considering the Java itself as a service, about profiles which mean, if I understand it well, some particular subset of functionality of the whole Java EE platform. Right now there is only web profile, but once the project Jigsaw is done, it should be more easier. With profiles it will come also the pruning of deprecated APIs such as XML-RPC, EJB CMP, JAXR, etc.

On the conference, I was attending only the EE stack; after the keynote there was some marketing talk of some Oracle guy about Oracle University project and the only thing I can recall is the discount 40% during the summer :]

The next presentation was about EclipseLink (previously called TopLink) which deals mostly with the ORM but they also introduced some new cool features I hadn’t known, for instance project MOXy. It is a JAXB provider utilizing the convention over configuration concept. In other words, no metadata is needed for describing the mapping between Java objects and XML for default behavior. For non standard things, the annotations can be used. The speaker was also showing some of the future features like mapping to JSON (one line change), direct mapping from the data from DB to XML/JSON and/or DBWS. DBWS is the RESTful api for relational DB. Even though it doesn’t make any sense to me from the Java perspective, it can be used, for instance, from HTML5+JavaScript in the future.

Next speaker covered the Jax-WS specification, mostly their own impl – Metro, nothing new. He kind of admitted, it is not so popular way of interoperability as it was. On the other hand all those WS-*, including security, reliability, etc., work well in a corporate environment where the systems have to trust each other. On the other hand building some client application is pain in comparison with REST. In other words, if you want to provide the API for community, the REST is much better.

Next talk was called¬†Java EE “hacking”. I used the quotes on purpose, because the guy was showing some wizzards in Netbeans IDE and using it he built some discussion forum in ~40 minutes. I wouldn’t call “clicking on wizzards” hacking, on the other hand it worked, there were many people present.

Last but one presentation was again the business one covering the Weblogic AS promo. It was very boring, but one idea was interesting for me. Using the¬†JSR 107¬†implementations (Java Cache: Infispan or Oracle’s one¬†for instance) for storing the users’ sessions and allow for better scalability, because the state is always the bottleneck, right? He outlined the interesting idea about failover in the heterogeneous¬†cluster of app servers using the described technique. The guy said “JBoss” three times, we are famousūüôā

Jax-RS was the last topic. It was about the news in 2.0 version i.e.¬†client API, content representation negotiation (from server side pov, if user doesn’t care in HTTP headers), Async calls, Hypermedia, etc. I liked this presentation the best.

At the end there were some free beer and music. To my surprise almost all topics (except few) were about open-source technologies, but it is somehow predicable to developer conference. I liked it, see you next year. By that time all the features will have been _almost_ done.

Did you like the JDD2012?

Tagged with:
Posted in Conferences, Java

Hello World

I am pleased to announce the release of my brand new blog 1.0 GA. Hopefully, it won’t die as all my previous attempts to blog.

I am going to write about Java, RHQ project, Metadata Editor, programming in general, my hobbies, etc.

Blogosphere, here I come!

Posted in Uncategorized
Follow

Get every new post delivered to your Inbox.