PFsense Firewall Visualization using Logstalgia

by on Oct.25, 2016, under Uncategorized

I’ve been playing with Logstalgia off and on for about a year now against my web hosts. It’s a fairly nifty way of monitoring the server and looking for errors. But, it’s not a truly effective tool in this context. I have always wanted to run it against my firewall. Unfortunately it’s not setup to parse other log types. It does however support an alternative log structure. This opens the door just a crack… just enough to let us force feed other datasources.

“Many other people” have done this according to posts I read. Using perl, some have supposedly setup syslog relays using the built-in perl libraries, or simply reformatted/reordered the logs through piping into Logstalgia. None have actually posted their solutions until now. Which is stupid.

I broke down and wrote my first perl script in over a decade. KISS is my motto, and it is reflected here. Feel free to improve. If there is sufficient interest, maybe I’ll add some more options. So, here is how I got mine working.

There are a number of things you need to do to set up your environment. I’m sorry if I have forgotten anything. This was built up over quite some time, and I’m mostly here to share the perl script and my workarounds.

1) Setup a second admin user on your pfsense. The root account does not land on the CLI when using ssh.
2) After you create the new admin account, make sure to grant it read write to the necessary files under /var/log/ (filter.log)

  • chmod +r /var/log/filter.log
  • 3) Setup ssh key authentication for the new admin account (you case safely skip this if you don’t know how)
    3) Install Perl if you don’t have it.
    4) Download Logstalgia
    5) Compile/install Logstalgia
    6) Save the script out to a working folder on your machine

    Here is the script:

    #!/usr/bin/perl -w

    my $filename = 'custom.log';
    while (<>) {
    open(my $fh, '>', $filename) or die "Could not open file '$filename' $!";
    my @array = split(/,/);
    my @datest = substr($array[1],0,15);
    print $fh time(); # date stamp
    print $fh "|$array[18]"; # Source IP
    print $fh "|$array[19]:$array[21]"; # dest IP:port
    print $fh "|$array[6]"; # action
    print $fh "|$array[17]"; # Size
    if( $array[6] eq 'block' ) {
    print $fh "|1"; # Block
    } else {
    print $fh "|0"; # Pass
    print $fh "\n";

    #print $fh "|$array[8]"; # color(IP4 vs IP6)
    #print $fh "|$array[16]"; # Referrer url(Protocol)
    #print $fh "|$array[]"; # User Agent
    #print $fh "|$array[]"; # Virtual Host
    #print $fh "|$array[3]"; # PID/Other(Rule Anchor)

    close $fh

    The script is executed by running:

    ssh admin-user@pf.local "clog -f /var/log/filter.log" | perl ./

    This will create a 1-line buffer file wherever you run the script. Clog is used to export out the current log buffer on the pfsense.

    After the script is running, tail pipe the buffer into Logstalgia:

    tail -f custom.log | logstalgia -sync

    Using -g you can customize your target addresses:

    tail -f custom.log | logstalgia -sync -g "Main Interface,URI=^,30" --no-bounce

    Have fun!

    1 Comment more...

    ZFS adventures

    by on Jul.30, 2012, under Uncategorized

    I have been playing with openindiana along with a few other platforms over the past few weeks, and thought I would share what I have discovered so far.

    Openindiana combined with napit turned out to be a very rudimentary system that was not what I was looking for. I played with it for a few days, but it just didn’t feel right to me. It was stable, but pretty much required me to manage the system over ssh. I eventually gave up and moved on to the derivatives.

    The first thine I tried was freenas. I had heard some good things about it, and the feature set looked solid. The overlap between what it offered and what I was looking for was pretty good. On initial installation, I went ahead and downloaded the latest release, 8.2. It installed fine, and booted up without issue. I configured my 3-drive pool and started testing speed and usability. And thats where things started to get weird. I kept getting strange situations where the gui would fall out of sync with the server, and I would have to reboot and clear my caches to get it working again. Looking online, this was confirmed in the forums, and appeared to be a common issue. The cavalier attitude of the developers towards the problem bothered me, but I kept going. After being satisfied that I wanted to move ahead with freenas, I exported my zfs pool on my machine, and moved the drives to freenas. Thats when the system fell to its knees. I spent two days working to try and figure out what was causing my issues, but I finally just gave up. I was never able to get the machine to boot with my 4 drives online. I even resorted to wiping the boot drive and starting again. During boot, at random places the machine would panic with memory access faults. I found that concept astounding in this day and age. Shaking my head I moved on.

    From there I moved on to Nexenta. It is also another openindiana derivative like Freenas. The licensing is much more strict and limiting. It definitely had the taste of vmware, and I was a bit hesitant.
    I went ahead and ran the install without issue. On reboot, the system loaded to a registration screen. Ugh. I had to “activate” my personal, home, free, hobby nas. Not impressed. I know everyone has to make money, but this is open source software. Activation is crap. At least it doesn’t appear to phone home constantly.
    I logged into the website, and entered my machine fingerprint, and got back the return code. Upon entering it into the server console, the load completed, and the basic server was online. The damn codes are horridly long as well. and since this is console, there is no cut and paste. Atrocious.
    The Nexenta by default boots to a random static IP, which is curious. All other systems I have used boot with DHCP or ask you what IP address to use. It was easily corrected with the console tools though, and we were off and running.
    In typical enterprise fashion, the system includes two wizards… because it just wouldn’t be enterprise ready without wizards! They walk you through naming your appliance and setting up the fundamentals.
    After the wizards completed, I went ahead and attempted to import my ZFS volume without issue. In order to setup shares and mount point on my pool, I had to upgrade it since the MacZFS software was so many versions behind. I decided to move ahead and do it, not being fully sure this was were I was going to end up.
    It ended up being a good decision. In spite of the typical, cliche, enterprise trappings, this seems to be a very solid platform. A NAS has to be reliable. Not just the drive system and data protection, but the underlying operation system as well. Its no good if the platform is panicking while you are trying to write to it.

    Over the weekend I put the system through its paces, and so far I am very pleased to with results, and frustrated by others. Nexenta seriously falls down when it comes to hardware monitoring and maintenance. It appears some to most of this is eliminated in the enterprise version, but I find that to be a poor excuse. Services and basic as SMART are not included or integrated, leaving you to depend on ZFS for error correction, prediction, and recovery. On the other hand, they do include many of their high end services such as data tiering and analytics. Its an exceptional tool.

    I’ll be hitting it hard going forward to see how it all shakes out.

    Leave a Comment more...

    Water cooling radiator fan trials

    by on Jul.30, 2012, under Uncategorized

    One of the great things about the water cooling kit was the fans. I didn’t think much of it at the time, but they were far superior to any other set of fans I had at the time. They had excellent static pressure and moved a good amount of air. But I didn’t think much of it until a bearing started occilating a few months back at medium speed. It actually sounded like a wire rubbing against the blades, but only when mounted blowing vertically up.

    I started looking through the many many online post about this subject matter, and began purchasing fans to see what I could come up with. After much trial and failure, I came to the same conclusion as a few other people out there. Three words. Scythe… Gentle… Typhoon. These are hard to find, but oh man what a fan. The hub is a lot bigger than I would like, but you can’t argue with the results. For the first time ever, I can actually feel air being blown through my radiator, and the temp difference has been dramatic as well. The idle temperature has dropped from 32 to 26 degrees. Basically the same temp as the air in the room.

    The first thing you notice when opening up an SGT, is the aggressive blade curvature. They are even more aggressive than my Cooler masters that have worked so well for me. The frame of the fan is much more sturdy than all others I have used. There is little to no give when mounting these into you case. The reason for this is probably the extremely tight clearance between the blades and the housing. Many of the other users had issues with noise, but in my tests they performed well.

    If you need a 120mm radiator fan, get an SGT 1850 RPM (D1225C12B5AP-15).

    Leave a Comment more...

    Water cooling phase 2

    by on Jul.14, 2012, under Uncategorized

    Adopting water cooling for my machine has been a great success. Over the past three years, I have had no failures of any part of the system.

    Things have aged though, and I have decided to replace a few worn out parts, and make some new additions. I seriously debated replacing the pump and reservoir, but just can’t justify the cost at this point. Both are working well… they just don’t look good. So instead I have decided to add a flow meter along with a control computer to better regulate the water temperatures.

    I decided on the Aquaero LT. Its pricier than the other options, but the capabilities are unmatched. I spent a good amount of time debating the three different models, but at 74, 149, and 249 dollars it was pretty much impossible to justify the more expensive options. I wanted a controller. I didn’t need a flashy gadget. The software that comes with the controller give you all of the capabilities regardless of model.

    For the flow meter, I picked up a Koolance INS-FM17N Unfortuantely, I didn’t get the matching Frequency Adaptor. I was hoping I didn’t need it, but at is turns out, there is no way to connect it to the controller. The Aquacomputer G1/4 Flow Meter Sensor Block is just a couple bucks more than the combined price. You would be better off just getting the integrated unit.

    At least now going forward if my pump fails, the machine can take action to protect itself and shut down. Its always concerned me.

    I bought everything through FrozenCPU again as you can tell, and they were as good as ever.

    Installation was about as complicated as you would expect. Its not for a novice, but if you have done any work with building machines, you should be fine. I was a bit taken aback by the lack of ESD protection in the packaging, but everything seems to be working fine.

    I plugged my fans into the fan headers on the board, plugged in the usb and power connections, then booted up the machine. The unit also supports controlling the pump as well, but I decided not to got there for now. It turned out to be a good idea. As the machine booted, I could see the fans were still on the radiator. Since I had a fresh load of fluid in the system, I was not overly concerned. it would take some time to heat everything up. Still, I expected the fan headers to default to full speed.

    After booting the machine, I loaded up parallels and ran the aqua suite 4 software. Nothing. I told it to go out and detect usb devices. Nothing. Checked for updates, and it found 4.72 was available. I went ahead and updated, but still got nothing, even though version 4 is labeled as supporting all current devices from Aqua Computer. Lost in translation I guess.

    I went to the website and downloaded version 2012 of the suite, and ran the installer. It completed fine, and upon first run I was presented with the option of updating my Aquaero LT. I went ahead and click the button, and it began updating me to v1022. After it completed, it told me to not load Aqua Suite until the device was reset. I sat there, waiting a bit with a blank screen. The device appeared to reboot, as it beeped. I loaded the software to see that I needed to do another firmware update. I was a bit frustrated as the Aqua Suite 2012 Software seemed to have no options available for configuration.

    I ran the update a second time, and allowed the thing to reset. This time it was different. As I loaded the software, tons of configuration options were presented to me in the console. I finally had a hold of the levers I was looking for.

    I have been testing quite a bi this evening, looking for the best fan curves to match my radiator and fan setup. So far I have come up with this:

    It allows my machine to run silent for almost all normal activities. When heavy loads are executed, the water temp increases, and the fans ramp up to increase the heat dissipation. This way the system only runs as loud as it needs to. Previously, the radiator fans were linked to the CPU header. This lead to full speed fans at the slightest load on the machine. That lead to excessive dust build up in the radiator, and more noise than was needed.

    I also discovered while working on the system that one of the bearings has begun to fail in one of my radiator fans. It runs fine in every position except the one I need it in. So next up is finding a new set of fans. We’ll see how that goes. For now I have one of my Noctua units in its place. I have always liked them, but the amount of air it moves it poor.

    Leave a Comment more...

    Looking for something?

    Use the form below to search the site:

    Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

    Visit our friends!

    A few highly recommended friends...