security career sickness

Do you ever suffer from client-induced nausea? You know, the feeling you get after the CISO jollily asks “How are we doing at security” soon after you’ve discovered they do next to NOTHING.

If you’re like me and if you get too many of these in short order you get sick. Sick like “I don’t care anymore, f*** this”. My nausea peaked recently and I wondered if complete cynicism is the unfortunate end.

I then re-read a snippet of an email I received:

“But they did hire you, so there must be a spark of initiative there.”

Yes. I was delighted to ponder this thought for a minute. Complete cynicism be damned!

How is it then you, I, anyone in security can keep healthy amidst what reality is like today? For me the answer is I like and have fun doing security. So to all my future gigs that make me sick, I don’t care, that is, I don’t care how bad or stupid it all is, security is a trip I like to be on.

Posted by tate Mon, 23 Aug 2010 14:59:00 GMT


pro tip: get lucky by scanning for 192.168.20.1

You’re in, but you don’t have God power yet. Before giving up, add a VIP and scan for 192.168.20.1.

“Your DRAC contains an integrated 10BASE-T/100BASE-T Ethernet NIC and supports TCP/IP. The NIC has a default address of 192.168.20.1 and a default gateway of 192.168.20.1.”

If that IP responds, you’re that much closer.

The default user name for this account is “root” and the default password is “calvin”.

Of course the above is well known and vulnerability scanners have checks for it. But as I just witnessed at a client, none of their historical vulnerability scan results discovered the cards because this client doesn’t use that IP block, yet several Dell servers had default DRAC cards waiting for some love. Be a good God.

Posted by tate Fri, 04 Jun 2010 20:54:00 GMT


pro tip: automating unix ossec agent installs using capistrano

If you like linux, capistrano, and ossec then you may like these scripts to automate installing ossec agents. I’m assuming you already have decent familiarity with ossec & capistrano.

prerequisites:

  • you can ssh to each of the hosts you want to install the ossec agent on using ssh pki (e.g. ssh-copy-id -i .ssh/id_rsa.pub alice@xxx.xxx.xxx.xxx)
  • you have nopasswd sudo privileges on each host (check via sudo -l)
  • you have a compiler (+ dependencies) installed on each host (yum -y install gcc glic-devel)
  • you have capistrano installed
  • you have the ossec source (current ver. = ossec-hids-2.4.1.tar.gz) (just put in your homedir)

steps you need to do:

  • uncompress the ossec source
  • use ossec-batch-manager.pl to quickly add agents to the server (located in the ossec contrib dir) (e.g. [alice@server]$ sudo ./ossec-batch-manager.pl -a -n host1 -p 10.10.10.10)
    note: you need to restart the ossec server after adding agents
  • copy ./contrib/specs/agent/preloaded-vars.conf to your homedir then configure it for how you want your agents to work (make sure you specify your USER_AGENT_SERVER_IP) - mine looks like:
    [alice@server]$ cat preloaded-vars.conf | egrep -v “^#|^$”
    USER_LANGUAGE=”en”
    USER_NO_STOP=”y”
    USER_INSTALL_TYPE=”agent”
    USER_DIR=”/var/ossec”
    USER_DELETE_DIR=”y”
    USER_ENABLE_ACTIVE_RESPONSE=”n”
    USER_ENABLE_SYSCHECK=”y”
    USER_ENABLE_ROOTCHECK=”y”
    USER_UPDATE=”y”
    USER_UPDATE_RULES=”y”
    USER_AGENT_SERVER_IP=”xxx.xxx.xxx.xxx”
  • create a file titled capfile in your homedir then copy & paste in the following code
    default_run_options[:pty] = true
    default_run_options[:max_hosts] = 25
    
    # VARIABLES
    set :ossec_version,     "2.4.1"
    
    # ROLES
    role :new_agents,
    '10.10.10.10',
    '10.10.10.11',
    '10.10.10.12'
    
    # TASKS
    # ================================================================================
    # Automatically install new UNIX based OSSEC agents  
    # ================================================================================
    namespace :new_agents do
    
      task :install_ossec_agent do
        upload("ossec-hids-#{ossec_version}.tar.gz", "/tmp/", :via => :scp)
        run "cd /tmp && tar zxf ossec-hids-#{ossec_version}.tar.gz"
        upload("preloaded-vars.conf", "/tmp/ossec-hids-#{ossec_version}/etc/preloaded-vars.conf", :via => :scp)
        run "cd /tmp/ossec-hids-#{ossec_version} && sudo ./install.sh"
        sudo "\\rm -rf /tmp/ossec-hids-#{ossec_version} && \\rm -f /tmp/ossec-hids-#{ossec_version}.tar.gz"
      end
    
      task :install_ossec_agent_keys do
        servers = find_servers_for_task(current_task)
         servers.each do |server|
          key = `sudo grep #{server} /var/ossec/etc/client.keys`
          logger.info "installing key on #{server}\n"
          put(key, "/tmp/client.keys", :hosts => server)
         end
        sudo "mv /tmp/client.keys /var/ossec/etc"
        sudo "chown root:ossec /var/ossec/etc/client.keys"
      end
    
      task :start_ossec_agents do
       sudo "/var/ossec/bin/ossec-control start"
      end
    
      task :do_all, :roles => :new_agents do
        install_ossec_agent
        install_ossec_agent_keys
        start_ossec_agents
      end
    
    end
    
        

  • modify the above code to match your environment (see below):
    # VARIABLES
    set :ossec_version,     "2.4.1" <---  put the version you downloaded here
    
    # ROLES
    role :new_agents,
    'xxx.xxx.xxx.xxx', <--- list the IPs of the hosts you want the ossec agent installed on
    'xxx.xxx.xxx.xxx',
    'xxx.xxx.xxx.xxx'  <--- no comma after listing the last IP
        

  • in your homedir (or wherever you have the ossec source, preloaded-vars.conf, & the capfile), execute the following to initiate scripted agent installation:
    [alice@server] cap new_agents:do_all

You should see lots of output and if the agents installed successfully then the tail end looks something like:

 ** [out :: 10.10.10.10] Starting OSSEC HIDS v2.4.1 (by Trend Micro Inc.)...
 ** [out :: 10.10.10.10] Started ossec-execd...
 ** [out :: 10.10.10.10] Started ossec-agentd...
 ** [out :: 10.10.10.10] Started ossec-logcollector...
 ** [out :: 10.10.10.10] Started ossec-syscheckd...
 ** [out :: 10.10.10.11] Starting OSSEC HIDS v2.4.1 (by Trend Micro Inc.)...
 ** [out :: 10.10.10.11] Started ossec-execd...
 ** [out :: 10.10.10.11] Started ossec-agentd...
 ** [out :: 10.10.10.11] Started ossec-logcollector...
 ** [out :: 10.10.10.11] Started ossec-syscheckd...
 ** [out :: 10.10.10.10] Completed.
 ** [out :: 10.10.10.11] Completed.
    command finished

bonus: here is a cap task to install syslog-ng from source minus the syslog-ng.conf file (you need to create that)

# =================================================================================================
# Automatically install syslog-ng from source
# =================================================================================================
task :install_syslog_ng, :roles => :new_agents do
  desc <<-DESC
  Disabling the default syslogd process on startup and stopping the current syslogd process
  DESC
  sudo "/etc/init.d/syslog stop"
  sudo "/sbin/chkconfig syslog off" 
  sudo "yum -y install gcc libnet" 
  run "wget https://www.balabit.com/downloads/files/syslog-ng/sources/2.1.4/source/syslog-ng_2.1.4.tar.gz --no-check-certificate" 
  run "wget https://www.balabit.com/downloads/files/eventlog/0.2/eventlog_0.2.9.tar.gz --no-check-certificate" 
  run "wget https://www.balabit.com/downloads/files/libol/0.3/libol-0.3.18.tar.gz --no-check-certificate" 
  run "\\rm -rf ./eventlog-0.2.9/" 
  run "tar zxf eventlog_0.2.9.tar.gz" 
  run "cd ./eventlog-0.2.9/;./configure" 
  run "cd ./eventlog-0.2.9/;make" 
  run "cd ./eventlog-0.2.9/;sudo make install" 
  run "\\rm -rf ./libol-0.3.18/" 
  run "tar zxf libol-0.3.18.tar.gz" 
  run "cd ./libol-0.3.18/;./configure" 
  run "cd ./libol-0.3.18/;make" 
  run "cd ./libol-0.3.18/;sudo make install" 
  run "\\rm -rf ./syslog-ng_2.1.4/" 
  run "tar zxf syslog-ng_2.1.4.tar.gz" 
  run "export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/;cd ./syslog-ng-2.1.4/;./configure" 
  run "cd ./syslog-ng-2.1.4/;make" 
  run "cd ./syslog-ng-2.1.4/;sudo make install" 
  desc <<-DESC
  creating the directory home to the syslog-ng.conf file
  DESC
  sudo "mkdir -p /usr/local/etc/syslog-ng" 
end

Posted by tate Sun, 16 May 2010 20:04:00 GMT


Microsoft may scan you

A client this month was observing strange FORM submittal activity and asked if I was still doing security testing of their web application.

Since I was not, I participated in investigating. I received a copy of their Apache log file.

I downloaded apache-scalp:

“Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET”

From viewing the output of Scalp! it was obvious an automated application scanner was being used. Three bits struck my curiosity:

  • The scanner was one I hadn’t heard of: netsparker
  • Netsparker is commercial, implying someone was willing to pay money to find vulnerabilities
  • The source of the scans was Microsoft

My client was dumbfounded why MS was doing this. I suggested they send an email to abuse@microsoft.com with the details and request their assistance.

Lo and behold, Microsoft replied and was responsible. Their Online Services Security and Compliance (OSSC) team was actively scanning, unbeknownst to my client .

Given this client has a relationship with MS, my guess is they signed something with fine print explicitly allowing MS to perform ad-hoc vulnerability assessments.

Good to know this can happen.

Posted by tate Sun, 31 Jan 2010 18:43:00 GMT


recap of a scope call

It’s been some time, but I just had one those scope calls from out of left field. University, mid size, private, budget sensitive, one security engineer and he is relatively new.

He is unsure of the state of his domain.
lost

Me: Have any security assessments ever been performed?
Client: No.
Me: pause Do you trust the integrity of your servers?
Client: No.
Me: pause You are processing credit cards?
Client: Yes.

Calls like this one ensue brain freeze in me. I get so hung up on them I mumble for a bit until I catch a clear thought.

On cue, this question soon followed: “I was thinking of starting with vulnerability assessments, but where do you recommend I start?”

Is this not a scenario security aficionados are to love?

I dread them now. I’m being tasked with teaching everything a functioning adult is to know on a phone call. Don’t get me wrong, I do love the flood of ideas this scenario stimulates, but I’m weary of working for clients so far behind in the game. My experience with like clients is they come to the play with little budget, support, and with even fewer resources.

But in the spirit of the holidays, I’m going to draft a list of tasks I’d recommend he do and I’ll share it here.

Posted by tate Wed, 30 Dec 2009 22:30:00 GMT


need to do a GET before POST, fuzzing with BURP and WebScarab

A recent web application I was testing was generating a unique hidden token on every render of the login page (revealed in the pink section in the screenshot below by WebScarab).

I had setup a BURP intruder run to iterate through a list of attack strings when I noticed a few of the responses contained a message displaying “unable to verify session”.

That is when I checked for a hidden value, and when discovered, guessed it was a dynamic key identifying each individual form submit.

The fuzzers in WebScarab and BURP typically work by repeatedly POSTing to a web form, like the login form above, then capturing and/or analyzing each response. Commercial scanners do the same. Repeatedly POSTing fails to work in this case.

Screenshot showing BURP’s intruder and that I want to run a sniper attack (i.e. send one attack string at this position (red text below) per POST request).

Screenshot showing a few custom attack strings loaded into BURP

Screenshot showing the execution and results of an intruder attack (i.e. looping through the attack string list and submitting each via a POST request)

Screenshot showing WebScarab’s fuzzer setup.

The problem in this scenario is the hidden token is only good for one POST request. Maybe I missed it, but I didn’t see an easy way to tell WebScarab’s fuzzer or BURP’s intruder to first perform a GET request (so I can capture a valid one time token value) then submit a POST request.

A few lines of Ruby along with the Mechanize library made for a simple work-around.

#!/usr/bin/ruby
require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
agent.set_proxy('localhost', '8008')

File.open("fuzzlist.txt") do |file|
  file.each_line {|fuzz_string| 
    agent.get('http://target/signin') do |page|
      form = page.forms.first 
      form.field_with(:name => 'data[user][email]').value = fuzz_string
      form.field_with(:name => 'data[user][password]').value = fuzz_string
      form.click_button
    end
  }
end

I configured mechanize to use my preferred proxy (BURP). The script loops through the fuzzlist.txt file, performs a GET request (now we have a valid token), then submits the form. Problem solved.

Posted by tate Thu, 19 Nov 2009 23:16:00 GMT


Signing things

I just saw that Thawte is ending their Web of Trust email keys program. I think they should have sold directory services in addition to providing keys, perhaps I'll write about that next. It got me thinking about SSL/TLS. I've always felt that the openssl program does a lot more than most people know to do with it. 

For example:

openssl dst -sha1 yourfile

is about 20% faster than "sha1sum yourfile" on my OpenSuse 10.1 AMD Opteron system. That might not be terribly interesting but 20% adds up on bigger files, like DVD images.

 Another thing is in this world of automatic updates and appstores, it's more important than ever to have a verification mechanism in place. OpenSSL provide all of the tools for some simple, yet strong, verification. A lot of people I know are confused by certificates and don't fully understand the different formats and what you can do with them but OpenSSL provides the raw basics to simply perform signing and verification.

Generate a 2048bit RSA key:

openssl genrsa -out rsapriv.pem 2048

Extract the public portion:

openssl rsa -in rsapriv.pem -pubout -out rsapub.pem

Sign a file and store the signature in testsig.bin:

openssl sha1 -sign rsapriv.pem -out testsig.bin yourfile

To verify:

openssl sha1 -verify rsapub.pem -signature testsig.bin yourfile

There you go, signing and verification without any of the "complicated" PKCS stuff. It's only marginally more work to actually add a CA to the mix.

Posted by Ian S. Nelson Wed, 07 Oct 2009 06:48:00 GMT


Man vs. web app

For me, it’s hard to downplay my joy from successfully assailing a web application and finding valuable faults: more so when nothing of value is reported from scanning using the most expensive app scanners.

On a recent gig I was focusing on finding issues with a couple of forms. Wielding BURP, I was intercepting all the GETs and POSTs and noticed the server responding with JSON data (i.e. http://[…]/data.json)

I tossed the JSON blob into an online JSON validator (http://www.jsonlint.com/) to make it easier to read (below is a small snippet).

"sanitized": 
  { aaaDetail": 
    { "aa_id": "276060", 
      "bb_id": "103065", 
      "cc_id": "515ad8933c821b2f72a8cbb7054zed3c", 
      "x_time": "2009-08-12 21:42:40",
      "y_time": "2009-08-12 21:47:34",
      "full_name": "T H", 
      "comment": null, 
      "country": "United States", 
      "state": "Colorado",
       "city": "Broomfield"
       [...]
     }
  } 

I soon realized the JSON objects contained great information for crafting precision attacks.

The JSON data reads like they are database column values. I wondered for a moment what would happen if I just use the strings in the JSON objects for POST keys, came up with logical values matching the keys, then started POSTing the modified form data.

Bingo. By simply intercepting POST requests and appending new key/value pairs to the list of key/value pairs already being submitted I was able to modify database values unattended by the developers. In this case, it meant I was able to modify information inserted and owned by other users of the site.

I’m not espousing Harry Potter skills here, rather illustrating one of many examples of why Man is needed to go beyond where automated web app scanners stop.

Posted by tate Sat, 22 Aug 2009 22:59:00 GMT


keep passphrases ‘in the mind’

I’ve always heard from friends that a court could compel a person to divulge a passphrase to get to their encrypted information.

I learned that is not exactly true while attending Tyler Pitchford’s presentation at Defcon 17.

Encryption keys are products of the mind and are protected by the Fifth Amendment.

The case from which these assertions are derived is United States v. Boucher. Sebastien may have to give up his passphrase in the end because he made a terrible decision in the beginning by consenting to talk.

Never talk. Never. If you haven’t already, watch this video presentation by Professor James Duane titled Don’t Talk to the Police to learn all about why.
Why I am proud to admit that I will never talk to any police officer. In Praise of the Fifth Amendment Right to Not Be a Witness Against Yourself.

Posted by tate Mon, 03 Aug 2009 12:13:00 GMT


tech notes on vSwitches and layer 2 attacks

I recently completed a pen test from the sole position of possessing remote administrative privileges to a few guest VMs.

I fired up yersinia to learn if launching layer 2 attacks could disrupt normal operations. While doing that I grabbed a copy of this book:
VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

vSwitches are reportedly immune to layer 2 attacks.

"Currently, a vSwitch will protect you from the following types of attacks by not providing the underlying functionality that these attacks require."
  • MAC Flooding
  • Double Encapsulation attacks (multiple 801.q envelopes)
  • Multicast Brute Force Attacks
  • Spanning Tree Attacks
  • Random Frame Attacks
  • DTP/VTP

I was able to perform MiTM via ARP cache poisoning on other visible guest VMs sharing the same network segment (because the attack targets the guest VM and not the vSwitch) – but in this case, each segment was allocated to a unique customer and therefore not a valuable test (i.e. poisoning one’s self was not in scope).

You can configure VMware to further limit attacks within a broadcast domain:

  • Security | Promiscuous Mode (default is to reject) : Reject
  • Security | MAC Address Changes (default is to accept): Reject
  • Security | Forged Transmits (allow outbound frame w/a source MAC that is different) (default is to accept) : Reject

Posted by tate Thu, 23 Jul 2009 15:47:00 GMT


Best to skip the pen test gigs with too short of attack windows

I just completed an external pen test whereby the rules of engagement limited the scan windows to two hours per night.  Requests for longer were rejected.

I hadn’t run within this tight of windows in some time and now I remember why I hate it so much.

I spent more time jacking any and every configuration setting I could tweak to boost each tool for balls out speed and baby-sitting (because failing seems to be a popular thing to do if you’re a tool sprinting at 50 threads and spending 0ms between requests) that I didn’t get nearly the time I wanted to concentrate on what I was paid to do: bust in.   

As a case in point I was working a SQLi point that was allowing me to download their entire database, alas, I only ever retrieved four of the 200+ tables during any one window.  Worse is I spun my wheels for several critical hours exerting fervent trial and error effort tweaking tool options, largely in vain, in hopes of making things go faster.  The consequence was tool tweaking dominated my attention.  Creativity, the force summoned for powersploiting, remained unconscious.    

 

Posted by tate Thu, 02 Jul 2009 22:27:00 GMT


Staffing and teams

I was listening to a podcast on Agile Staffing last week and they succinctly stated a couple things everyone sort of knows but doesn't say that often.
  • On small startup type teams, the team is everything, a bad team member can kill the product and the company.
  • On particularly agile teams, having agile people is better than having technology experts.
The first point is really clear, be it the leadership of the team or contributors to the team, if any one piece is broken then the whole thing won't work well. The second point is something I think people tend to dismiss, people like to list desired skills in job descriptions more so than desired attitudes and in an interview it's far more likely you'll be asked to write some code or explain some sort of process than you will be given a personality profile. One pattern that I've seen at a number of companies I've worked at is that new people will be some how challenged and asked to do some heroic amount of work and the company sort of over reaches. After that challenge project is done the team never fully recovers, the team is skeptical of everything and doesn't want to work that hard again. All future projects are exercises in work reduction, not so much in product improvement. The team is reluctant to do anything like the challenge again. You end up with something that's fundamentally not repeatable. Even if you end up with a great result, the team is fried and you can't follow it up. Another pattern I've seen is the so called "analysis paralysis." The desire to find a singular, "perfect," solution to a problem outweighs the desire to do anything else. Rather than attempting to fix problems or "solve the problem multiple times," or put "band-aids" on problems there is a desire to wait until an ultimate solution is created, which is usually never. With the problem, there are usually some fairly easy things that can be done to make some sort of incremental improvements along the way. Back to that second point, there are personality traits that help you find people that help you break those patterns. There are people that are willing to iterate on solutions and try to repeat success and those are the people you want to put on teams. Now this is all software talk but does it apply to security and network teams? Is a security policy something that has to be iterated on and changed over time or is there a "perfect" solution that can be reached? You can have the best security guys in the world working for you but if you've overextended them do they still actually work?

Posted by Ian S. Nelson Tue, 30 Jun 2009 10:28:00 GMT


Defending data? Incentivize.

Defending boils down to skill and incentive.

Tools that provide visibility are required, but that’s another topic. Skill is also an obvious requisite.

How about incentives? What incentives are in place for security engineers to really dig in and be great defenders?

Tier one engineers working out of traditional MSSPs are paid in the $20 per hour range or lower and by the nature of their position they have minimal understanding of anything of substance about their clients’ networks.

Opposite that, I know a very talented group of engineers working at an expensive outsourced IT/software company whom are responsible for their company’s top paying customers. They breathe uptime. Heads roll and the company loses money when downtime occurs. Security is barely an afterthought. Lest they do see a security issue, they may skip notifying for why create more support tickets and work.

It doesn’t make sense to punish defenders for failing to prevent infiltration – that is to be expected today. Simply detecting one is a great accomplishment.

To move defenders to be great defenders, reward them for detecting infiltrations.

I think it would be great for company’s to hire an outside party to perform unauthorized activity at an increasing pace and breadth until someone responsible for monitoring sounds the alarm. Reward those who discovered the activity. Do it frequently. Change it up. Make a competition out of it. No doubt this would help weed out bad performers, be they internal or external.

This is very similar to what I’ve seen a few hospitals implement. If an employee stops and challenges a person without a badge then they receive a $100 bonus.

I’m not promoting pen. testing here, though it’s a good example of a challenge.  I think simple and frequent small scale tests tied to rewards would work wonders for many security groups and for the company’s wanting to keep their assets protected.

Posted by tate Sun, 31 May 2009 12:41:00 GMT


frustrating log forwarding issue, resolved it today!

I ran into a technical logging obstacle and I finally passed it today.

Setup:

  • A large number of linux servers deployed over disparate locations and connected via consumer cable and xDSL lines
  • All of them are configured to forward logs to a remote central log server via syslog-ng and UDP (client prefers UDP over TCP)

Problem:

  • We were noticing a growing percentage would stop forwarding logs to the central log server

Troubleshooting notes:

  • Timing seemed random; many clients forwarded logs successfully for weeks or months, some clients would stop forwarding after a few weeks, a few would stop in a couple of days
  • All other connectivity appeared fine when troubleshooting (e.g. ssh worked, http worked, icmp, etc.)
  • On each server, syslog-ng was running, logging locally fine, and showed no error conditions (but we observed no outbound traffic when executing tcpdump port 514)
  • We did notice if we restart syslog-ng, forwarding would work again and we would immediately observe new logs appearing on the central log server

Quick thoughts and things we tried:

  • Okay, configs are likely good because all works when restarting
  • Network issues? But then why does restarting syslog-ng have an immediate and somewhat lasting effect?
  • Alright, let’s upgrade to the lastest sylog-ng 2.x suite (libol, eventlog, syslog-ng) because maybe we’re running into a weird bug

Finali:

  • After upgrading a dozen or so and waiting a week, we noticed two or three stopped forwarding logs again
  • We executed tcpdump port 514 and again observed no outbound traffic
  • We restarted syslog-ng and as before, observed new logs appearing on the central log server. This time it caught our attention that our tcpdump window still showed no outbound traffic even after the syslog-ng restart. Ah, now we’ve identified the information we were neglecting to consider: multiple interfaces.

[tt@00-e0-4d-8d-ce-3b ~]$ /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr
[]
tun0 Link encap:UNSPEC HWaddr
[]

We executed tcpdump -i tun0 port 514 on a server for which we had not observed log messages on the central log server in some time. By watching tun0, we now observed outbound UDP packets, but the central log server was still not receiving them. When we restarted syslog-ng on this server, we now noticed a significant change in the tcpdump output. The first three lines below were observed when the central syslog server was not receiving messages and the second three lines are after we restarted syslog-ng.

15:57:22.009114 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG authpriv.notice, length: 132
15:57:22.187959 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.info, length: 104
15:57:22.188023 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.notice, length: 97
15:57:22.699615 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 249
15:57:22.700051 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 254
15:57:22.700108 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 246

Problem discovered (some of the IP information above was changed for obvious reasons).

On reboot or startup, tun0 is brought up and syslog-ng forwarded logs with the correct source IP for this environment (the one assigned to tun0). The catch is when an openvpn connection would bounce on a live system (i.e. tun0 would go down), syslog-ng would fall back to using the source IP assigned to eth0 and continued to use that IP even after tun0 was brought back up, which broke the forwarding of logs in this environment. Syslog-ng required a restart after tun0 would go down in order to use the proper IP (as observed in the tcpdump output for tun0 above). The flapping openvpn connection was the culprit.

Posted by tate Mon, 20 Apr 2009 22:04:00 GMT


Need to create a quick script to detect if a credit card number pattern is being written to any file on a Linux server?

Try inotify.

Inotify is a Linux kernel feature that monitors file systems and immediately alerts an attentive application to relevant events, such as a delete, read, write, and even an unmount operation.

The Inotify-tools library provides a pair of command-line utilities to monitor file system activity:

  • inotifywait simply blocks to wait for inotify events. You can monitor any set of files and directories and monitor an entire directory tree (a directory, its subdirectories, its sub-subdirectories, and so on). Use inotifywait in shell scripts.
  • inotifywatch collects statistics about the watched file system, including how many times each inotify event occurred.

We just need to check two events:

  • # IN_CREATE - File/directory created in watched directory
  • # IN_MODIFY - File was modified

#!/usr/bin/ruby
require 'inotify'
require 'find'

raise("Specify a directory") if !ARGV[0]
directory = ARGV[0]

i = Inotify.new

t = Thread.new do
      i.each_event do |event|
        File.open(directory + "/#{event.name}") do |f|
          f.grep( /\b(?:\d[ -]*){13,16}\b/) do |line|
            puts "Detected credit card pattern in directory #{directory}, file #{event.name}\n"
          end
        end
      end
end

Find.find(directory) do |e|
  begin
    i.add_watch(e, Inotify::CREATE | Inotify::MODIFY)
  rescue
    puts "Skipping #{directory}: #{$!}"
  end
end

t.join

Not too bad for an hour of playing around. It works.

I used the ruby inotify version 0.0.2 from http://raa.ruby-lang.org/project/ruby-inotify/, but if you do that, then you need to fix line 47 - change it to

r = rb_thread_select (fd+1, &rfds, NULL, NULL, NULL);
as documented here: http://www.mindbucket.com/2009/02/24/ruby-daemons-verifying-good-behavior/. The code above I modified from the example included with ruby inotify.

Posted by tate Wed, 01 Apr 2009 01:45:00 GMT


removed for the time being

removed for the time being...

Posted by tate Sat, 28 Feb 2009 16:29:00 GMT


Insects

A friend told me recently his new security hire has two passions in life: security and poker. The word passion is strong and I can imagine this new hire’s use of that word and his subsequent explanation helped him to win the position. If his win was from describing his poker passion, then that’s cool and I have nothing to add.

Anyhow, this made me wonder if there's an area of security I could say I'm currently passionate for. One area did stand out and the distinction I made was between offensive and defensive security.

For me, defensive security is it.

I’m not convinced my career survival instincts are somehow overpowering my reason, but to me it feels like to be awesome at offensive security is to be an insect. Thinking back over several penetration tests I clearly remember wishing I had more knowledge of the workings of a particular exposed service or application.

This is not to say it’s not fun to be an insect at the right time. With penetration testing, it can be crazy fun to be an insect, surrounding by lots of other insects of different skills, and all attacking the same target with abandon. It’s not though when you run with a small crew and run into a service or app for which no one has familiarity.

Defense on the other hand is a lot less insect like. You can be really good yet skip out on the training for your newly deployed app server. Playing defense therefore is an evolutionary step up of sorts and requires more intelligence that playing the offensive insect (pun intended).

Ha. Only kidding.

My take is a small and intelligent team can play defense very well without the need to dive deep into their infrastructure's every exposure. That is why I like defense - you can grasp its entirety.

Posted by tate Mon, 26 Jan 2009 01:44:00 GMT


Know thy network well

There is a good newsgroup thread running on Dailydave.
http://seclists.org/dailydave/2008/q4/0085.html

The takeaways are familiar reminders for the cognoscenti, but it’s still good to read and good for referencing.

“Patch management, IDS, Anti-Virus, scanners of all shapes and sizes. Audits” don’t work against competent attackers.

“And they [Penetration testers] agreed on two things: the threats you know about are not the ones you need to worry about; and every network is own-able. Every. Single. One.”

“If you accept the premise that it's not possible to protect every asset (or even protect any single asset completely), then the logical action is to identify the most valuable assets and secure them to the best of your ability” column by Dennis Fisher

I'm a big fan of using tools that help me get visibility into what is happening on a network, which is why I like these statements:

“Baseline system and network behaviour. Analyse any abnormal behaviour. (Easier said than done. You may never see anything.)” (raus)

“I would also note that it's misleading to say you should throw in the towel because one unpublished vuln can pop your box. There is more to it than that if you are doing your job right. Can they pop it without being discovered... for how long, and how often?” (Dragos Ruiu")

Marc Maiffret opined:

“The biggest threat to the average computer user is not zeroday vulnerabilities but system misconfigurations and vulnerabilities within third party applications. Most organizations are only just starting to get a handle on patching Microsoft vulnerabilities let alone third party applications. This becomes even more apparent with consumers and small to medium sized businesses where they only have Windows Update and WSUS to depend on. There is simply no third party patching being done in these environments making it a LOT more likely for them to get owned with a 6 month old Adobe Acrobat vulnerability than some zeroday vulnerability. This is currently the lowest hanging fruit for attackers and does not require an attacker to have large sums of money to waste on buying zeroday attacks.

It’s clear security teams must deploy tools that add to their sense of understanding for what is normal activity. You want intuition and clarity. You want to have that gut-instinct and confidence that you can detect if something is not quite right. The way to do that is to deploy tools that enchance visibility (i.e. tools that show you traffic patterns and volumes, running applications, logins, tools that point out unusual activity, etc.).

Posted by tate Sat, 13 Dec 2008 23:20:00 GMT


popular clearnet blog entries from the past

Posted by tate Sat, 06 Dec 2008 09:03:00 GMT


this will fuel paranoia

How soon will we see crazy apps built on top of something like macrosense? The more wile and guile types will welcome this to sweeten their malevolence.

  • "What if you could look at your cell phone and see a heat map of where everybody in the city was at that very moment?"
  • "Over time, it learns about where you like to go (fancy restaurants or punk rock clubs) and shows you other people like you, and where they are—right now."
  • "Imagine a GPS that didn’t just tell you where you are or where your programmed destination is, but a GPS in your phone that actually predicts what you want to do and where you want to go."

Yeah, it’ll be wonderful when you’re driving along and the GPS announces the strip club is up on the right when your wife is riding along. Honey, I have no idea why the GPS thinks I want to go there. Stupid GPS.

News article: http://www.iht.com/articles/2008/06/22/business/22proto.php

Posted by tate Wed, 19 Nov 2008 04:28:00 GMT