mobilexco/laravel-scout-elastic; an AWS Elasticsearch driver for Laravel Scout

A large piece of what we’ll be doing the last half of this year is improving the support workflows inside Tether (our MarTech platform) and that includes Search. Being a Laravel shop, it makes sense to start with Scout to see if that gets us close, if not completely over the line.

We use Elasticsearch for other things in Tether so it made sense to use that as the Scout backend through the super helpful ErickTamayo/laravel-scout-elastic package. And it worked as advertised right out of the box for local development (inside Vagrant with a local Elasticsearch server). But as soon as we moved the code to a cloud environment that used an AWS Elasticsearch instance it was throwing all sorts of wacky errors. Turns out, AWS Elasticsearch is different than Elastic Elasticsearch — not completely, just how communication is sent over the wire.

No problem. we’re clearly not the first one to discover this problem, and sure enough there are a number of forks of the package that add this in. But, they commit one of the following sins;

  • Required AWS credentials to be checked into your repo in a .env file
  • Used env() to fetch configuration settings which breaks if you are caching configs (and you really should be)
  • Required manual intervention with Elasticsearch while deploying.

These are the result of Laravel being in the awkward teenage years, and all very solvable. I just wish it didn’t seem that things I need require these fixes…

Anyhow, mobilexco/laravel-scout-elastic uses the defaultProvider() which means it will work with IAM roles on your AWS infrastructure to authenticate with Elasticsearch. This the official AWS recommended approach and does not require the presence of keys on the server (and all the pain around rotation, etc. that comes with using keys).

It also publishes conf/laravel-scout-elastic.php for setting flags it needs to decide whether to use Elastic or AWS implementations of Elasticsearch rather than env() so config:cache works. (This should likely be better called out in the Laravel docs for creating packages…)

The package also includes a new Artisan command (scout:create-index) which can be called by via something like AWS CodeDeploy (in the AfterInstall hook) to ensure the index is created that Scout will be using. Which is useful if you are in an environment where your Elasticsearch access is restricted to only the boxes that need access and those boxes don’t have ssh installed on them. (Artisan commands are run either CodeDeploy or SSM.)

Hopefully this saves someone the 12 hours of distracted development it took to come up with this solution.

Client-specific domains with CloudFormation for clients that use Google as email provider

A number of our clients want vanity domains for their experiences, which adds a laywer (or two) of operations overhead beyond just having a line item in the invoice. In the spirit of ‘infrastructure as code’-all-the-things, this is now my process for registering new domains for our clients

  1. Register the domain through Route 53
  2. Delete the hosted zone that is automatically created. (It would be nice if there was an option when getting the domain to not automatically create the hosted zone.)
  3. Login to Google Apps and add the domain as an alias. When prompted to verify, choose Ghandi as the provider and get the TXT record that is needed
  4. Create a CloudFormation stack with this template. Some interesting bits;
    • Tags in the AWS Tag Editor are case sensitive so ‘client’ and ‘Client’ are not equivilent
    • I think all my stacks will include the ‘CreationDateParameter’ parameter from now on which gets added as a tag to the Resource[s] that can accept them. This is part of the ‘timebombing’ of resources to make things more resilient. In theory I can also use AWS Config to find Resources that are not tagged and therefore presumably under CloudFormation control.
    • Same thing for the ‘client’ tag. Though still nto keen on that name or billing_client or such.
    {
      "AWSTemplateFormatVersion": "2010-09-09",
      "Parameters": {
        "ClientNameParameter": {
          "Type": "String",
          "Description": "Which client this domain is for"
        },
        "DomainNameParameter": {
          "Type": "String",
          "Description": "The domain to add a HostedZone for"
        },
        "GoogleSiteVerificationParameter": {
          "Type": "String",
          "Description": "The Google Site Verification TXT value"
        },
        "CreationDateParameter" : {
          "Description" : "Date",
          "Type" : "String",
          "Default" : "2017-08-27 00:00:00",
          "AllowedPattern" : "^\\d{4}(-\\d{2}){2} (\\d{2}:){2}\\d{2}$",
          "ConstraintDescription" : "Date and time of creation"
        }
      },
      "Resources": {
        "clienthostedzone": {
          "Type": "AWS::Route53::HostedZone",
          "Properties": {
            "Name": {"Fn::Join": [".", [{"Ref": "DomainNameParameter"}]]},
            "HostedZoneTags": [
              {
                "Key": "client",
                "Value": {"Ref": "ClientNameParameter"}
              },
              {
                "Key": "CloudFormation",
                "Value": { "Ref" : "CreationDateParameter" }
              }
            ]
          }
        },
        "dnsclienthostedzone": {
          "Type": "AWS::Route53::RecordSetGroup",
          "Properties": {
            "HostedZoneId": {
              "Ref": "clienthostedzone"
            },
            "RecordSets": [
              {
                "Name": {"Fn::Join": [".", [{"Ref": "DomainNameParameter"}]]},
                "Type": "TXT",
                "TTL": "900",
                "ResourceRecords": [
                  {"Fn::Sub": "\"google-site-verification=${GoogleSiteVerificationParameter}\""}
                ]
              },
              {
                "Name": {"Fn::Join": [".", [{"Ref": "DomainNameParameter"}]]},
                "Type": "MX",
                "TTL": "900",
                "ResourceRecords": [
                  "1 ASPMX.L.GOOGLE.COM",
                  "5 ALT1.ASPMX.L.GOOGLE.COM",
                  "5 ALT2.ASPMX.L.GOOGLE.COM",
                  "10 ALT3.ASPMX.L.GOOGLE.COM",
                  "10 ALT4.ASPMX.L.GOOGLE.COM"
                ]
              }
            ]
          }
        },
      }
    }
  5. Update the domain’s nameservers for the ones in our newly created Hosted Zone. I suspect this could be done via a Lambda backed custom resource, but that’s a couple steps too complicated for me right now. If I have to do this more than once every couple weeks it’ll be work the learning time.
  6. Validate the domain with Google.
  7. Manually create a certificate for ${DomainNameParameter} and *.${DomainNameParameter}. (For reals, this should be an automatic thing for domains registered in Route 53 and hosted within Route 53.)

And then I need to create an ALB for the domain and point it at the right service. But thats getting rather yak shave-y. The ALB needs to be added to the ASG for the service but those are not under CloudFormation control so I need to get them under control.

HubSpot in an AWS World

We recently moved our corporate website from WPEngine to HubSpot and as part of that, you have to do some DNS trickery. HubSpot helpfully provides instructions for various DNS providers, but not Route 53. Reading the ones they do provide though provides a good idea what is needed;

  1. Add a CNAME for your HubSpot domain as the www record
  2. Add an S3 hosting bucket to redirect everything to www.yourdomain.com
  3. Add a CloudFront distribution to point to your bucket

Now, this is likely 5 minutes of clicking, but AWS should be done with minimal clicking, in favour of using CloudFormation (or TerraForm or such). As such, it took about 10 hours…

Lesson 1 – Don’t create your Hosted Zones by hand.

Currently, all our Hosted Domains in Route 53 were either created by hand as the domains were registered somewhere else, or created at registration time by Route 53. This is a challenge as CloudFormation cannot edit (to add or update) records in Hosted Zones that were not created by CloudFormation. This meant I needed to use CloudFormation to create a duplicate Hosted Zone and let that propagate through the internets and then delete the existing one.

Here’s the CloudFormation template for doing that — minus 70+ individual records. Future iterations likely would have Parameters and Outputs sections, but because this was a clone of what was already there I just hardcoded things.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "zonemobilexcocom": {
      "Type": "AWS::Route53::HostedZone",
      "Properties": {
        "Name": "mobilexco.com."
      }
    },
    "dnsmobilexcocom": {
      "Type": "AWS::Route53::RecordSetGroup",
      "Properties": {
        "HostedZoneId": {
          "Ref": "zonemobilexcocom"
        },
        "RecordSets": [
          {
            "Name": "mobilexco.com.",
            "Type": "MX",
            "TTL": "900",
            "ResourceRecords": [
              "1 ASPMX.L.GOOGLE.COM",
              "5 ALT1.ASPMX.L.GOOGLE.COM",
              "5 ALT2.ASPMX.L.GOOGLE.COM",
              "10 ALT3.ASPMX.L.GOOGLE.COM",
              "10 ALT4.ASPMX.L.GOOGLE.COM"
            ]
          }
        ]
      }
    },
    "dns80808mobilexcocom": {
      "Type": "AWS::Route53::RecordSetGroup",
      "Properties": {
        "HostedZoneId": {
          "Ref": "zonemobilexcocom"
        },
        "RecordSets": [
          {
            "Name": "80808.mobilexco.com.",
            "Type": "A",
            "TTL": "900",
            "ResourceRecords": [
              "45.33.43.207"
            ]
          }
        ]
      }
    }
  }
}

Lesson 2 – Don’t forget that DNS is all about caching and you could clone a domain and forget to include the MX record because you blindly trusted the output of CloudFormer only to realize you had stopped incoming mail overnight but worked for you because you had things cached…

Lesson 3 – Even though you are using an S3 Hosted Website to do the redirection, you are not actually using an S3 Hosted Website in the eyes of Cloud Front.

This cost me the most amount of grief as it led me to try and create an S3OriginPolicy, an Origin Access Identity, etc. that I didn’t need.

Note: in order to make this template work, you need to first have issued a certificate for your domain through ACM. Which is kinda a pain. My current top ‘AWS Wishlist’ item is auto-provisioning of certificates for domains that are both registered and hosted within your account.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Parameters": {
    "DomainNameParameter": {
      "Type": "String",
      "Description": "The domain to connect to hubspot (don't include the www.)"
    },
    "HubspotCNameParameter": {
      "Type": "String",
      "Description": "The CName for your hubspot site"
    },
    "AcmCertificateArnParameter": {
      "Type": "String",
      "Description": "ARN of certificate to use in ACM"
    }
  },
  "Resources": {
    "s3mobilexcocom": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketName": {"Ref": "DomainNameParameter"},
        "AccessControl": "Private",
        "WebsiteConfiguration": {
          "RedirectAllRequestsTo": {
            "HostName": {"Fn::Join": ["", ["www.", {"Ref": "DomainNameParameter"}]]},
            "Protocol": "https"
          }
        }
      }
    },
    "dnswwwmobilexcocom": {
      "Type": "AWS::Route53::RecordSetGroup",
      "Properties": {
        "HostedZoneId": {
          "Fn::ImportValue" : "hosted-zone-mobilexco-com:HostedZoneId"
        },
        "RecordSets": [
          {
            "Name": {"Fn::Join": ["", ["www.", {"Ref": "DomainNameParameter"}, "."]]},
            "Type": "CNAME",
            "TTL": "900",
            "ResourceRecords": [
              {"Ref": "HubspotCNameParameter"}
            ]
          }
        ]
      }
    },
    "dnsmobilexcocom": {
      "Type": "AWS::Route53::RecordSetGroup",
      "Properties": {
        "HostedZoneId": {
          "Fn::ImportValue" : "hosted-zone-mobilexco-com:HostedZoneId"
        },
        "RecordSets": [
          {
            "Name": {"Fn::Join": ["", [{"Ref": "DomainNameParameter"}, "."]]},
            "Type": "A",
            "AliasTarget": {
              "DNSName": {"Fn::GetAtt": ["httpsDistribution", "DomainName"]},
              "HostedZoneId": "Z2FDTNDATAQYW2"
            }
          }
        ]
      }
    },
    "httpsDistribution" : {
      "Type" : "AWS::CloudFront::Distribution",
      "Properties" : {
        "DistributionConfig": {
          "Aliases": [
            "mobilexco.com"
          ],
          "Origins": [{
            "DomainName": {"Fn::Join": ["", [{"Ref": "DomainNameParameter"}, ".s3-website-", {"Ref": "AWS::Region"}, ".amazonaws.com"]]},
            "Id": "bucketOriginId",
            "CustomOriginConfig": {
              "HTTPPort": 80,
              "HTTPSPort": 443,
              "OriginProtocolPolicy": "http-only"
            }
          }],
          "Enabled": "true",
          "DefaultCacheBehavior": {
            "ForwardedValues": {
              "QueryString": "false"
            },
            "TargetOriginId": "bucketOriginId",
            "ViewerProtocolPolicy": "allow-all"
          },
          "ViewerCertificate": {
            "AcmCertificateArn": {"Ref": "AcmCertificateArnParameter"},
            "SslSupportMethod": "sni-only"
          },
          "PriceClass": "PriceClass_100"
        }
      }
    }
  }
}

Lesson 4 – Naming conventions are a thing. Use them.

As soon as you start doing ImportValue or AWS::CloudFormation::Stack. In theory the ImportValue lines could use DomainNameParameter with Fn::Sub to switch the . to a – and this would be an entirely generic template, but this is working well enough for me. And of course, your naming convention could be (and likely is) different.

Harmonizing Maintenance Windows

At the moment we are only using RDS and ElastiCache within AWS, but more services we use the more maintenance windows is going to come up. Rather than have them at random places around the week and clock, I figure it would be useful to have just a single window that we can subsequently work into our SLAs etc. Now, I really like the management consoles AWS has, but its a lot of clicks to track things — especially if I start using something like CloudFormation and Autoscaling or something to be making things magically.

Scripting to the rescue.

Our applications are PHP based, but at my heart I’m a Python guy, so I whipped up one. And aside from the fear of modifying running items, it appears to have worked well.

import boto3
 
maintenance_window = 'sun:09:35-sun:10:35'
 
# rds can have maintenance windows
update_rds = False
rds = boto3.client('rds')
paginator = rds.get_paginator('describe_db_instances')
for response_iterator in paginator.paginate():
    print('Current RDS Maintenance Windows')
    for instance in response_iterator['DBInstances']:
        print('%s: %s UTC' % (instance['DBInstanceIdentifier'], instance['PreferredMaintenanceWindow']))
        if instance['PreferredMaintenanceWindow'].lower() != maintenance_window.lower():
            update_rds = True
 
if update_rds == True:
    paginator = rds.get_paginator('describe_db_instances')
    for response_iterator in paginator.paginate():
        for instance in response_iterator['DBInstances']:
            if instance['PreferredMaintenanceWindow'].lower() != maintenance_window.lower():
                rds.modify_db_instance(
                    DBInstanceIdentifier=instance['DBInstanceIdentifier'],
                    PreferredMaintenanceWindow=maintenance_window
                )
    paginator = rds.get_paginator('describe_db_instances')
    for response_iterator in paginator.paginate():
        print('Adjusted RDS Maintenance Windows')
        for instance in response_iterator['DBInstances']:
            print('%s: %s UTC' % (instance['DBInstanceIdentifier'], instance['PreferredMaintenanceWindow']))
 
# elasticache can have maintenance windows
update_ec = False
ec = boto3.client('elasticache')
paginator = ec.get_paginator('describe_cache_clusters')
for response_iterator in paginator.paginate():
    print('Current ElastiCache Maintenance Windows')
    for instance in response_iterator['CacheClusters']:
        print('%s: %s UTC' % (instance['CacheClusterId'], instance['PreferredMaintenanceWindow']))
        if instance['PreferredMaintenanceWindow'].lower() != maintenance_window.lower():
            update_ec = True
 
if update_ec == True:
    paginator = ec.get_paginator('describe_cache_clusters')
    for response_iterator in paginator.paginate():
        for instance in response_iterator['CacheClusters']:
            if instance['PreferredMaintenanceWindow'] != maintenance_window:
                ec.modify_cache_cluster(
                    CacheClusterId=instance['CacheClusterId'],
                    PreferredMaintenanceWindow=maintenance_window
                )
 
    paginator = ec.get_paginator('describe_cache_clusters')
    for response_iterator in paginator.paginate():
        print('Adjusted ElastiCache Maintenance Windows')
        for instance in response_iterator['CacheClusters']:
            print('%s: %s UTC' % (instance['CacheClusterId'], instance['PreferredMaintenanceWindow']))

It’s always a Security Group problem…

I’ve got a number number of private subnets within my AWS VPC that are all nice and segregated from each other. But every time I light up a new Ubuntu instance and tell it to ‘apt-get update’ it times out. Now, since these are private subnets I can get away with opening ports wide open, but AWS is always cranky at me for doing so. I feel slightly vindicated that the same behaviour is asked about on Stack Overflow often too, but anyways, I figured it out this week. Finally. And as usual with anything wonky network-wise in AWS it was a Security Group problem.

  1. First thing, read the docs carefully.
  2. Read it again, more careful this time
  3. Setup the Routing. I actually created 2 custom routing tables rather than modify the Main one; explicit is better than implicit (thanks Python!)
  4. Create an ‘apt’ Security Group to be applied to the NAT instance with the inbound rule, from your private VPC address space for HTTP (80), HTTPS (443) and HKP (11371). HTTP is the default protocol for apt but if you are adding new repos the key is delivered via HTTPS and then validated against the central key servers via HKP. You’ll need outbound rules for those ports too per the docs

And now you should be able to lock down your servers a bit more.

Faster feedback by limiting information frequency

Code coverage is one of those wacky metrics that straddles the line between useful and vanity. On one hand, it gives you an idea of how safely you can make changes, but on the other it can be a complete fake-out depending on how the tests are constructed. And it can slow your build down.

A lot.

I suspect a lot of our pain is self-induced, but our Laravel application’s ‘build and package’ job jumps from under 2 minutes to around 15 once we turn on code coverage. Ouch.

So I came up with a compromise position in the build in that the tests always run, but the coverage only gets calculated every 15th build (around once a day). Here is what the relevant task for that job now looks like.

# hack around jenkins doing -xe
#set +e
 
mkdir -p jenkins/phpunit
mkdir -p jenkins/phpunit/clover
 
# run coverage only every 15 builds
if [ $(($BUILD_ID%15)) -eq 0 ]; then
  phpunit --log-junit jenkins/phpunit/junit.xml --coverage-clover jenkins/phpunit/clover.xml --coverage-html jenkins/phpunit/clover
else
  phpunit --log-junit jenkins/phpunit/junit.xml
fi
 
# hack around presently busted test
#exit 0

Some things of note;

  • The commented out bits at the beginning and end allow me to force a clean build if I really, really want one
  • My servers are all Ubuntu so use ‘dash’ as its shell which forces slightly different syntax which my fingers never get right the first time
  • I don’t delete the coverage log so the later ‘publish’ action doesn’t fall down. It just republishes that again
  • As we hire more people and the frequency of things landing in the repo increases, I’ll likely increase the spread from 15 to something higher
  • At some point we’ll spend the time to look at why the tests are so slow, but not now.

Using Puppet to manage AWS agents (on Ubuntu)

One of the first thing any cloud-ification and/or devops-ification project needs to do is figure out how they are going to manage their assets. In my case, I use puppet.

AWS is starting to do more intensive integrations into things using agents that sits in your environment. This is a good, if not great, thing. Except if you want to, oh, you know, control what is installed and how in said environment.

Now, it would be extremely nice if AWS took the approach of Puppet Labs and host a package repository which would mean that one could do this in a manifest to install the Code Deploy Agent.

  package { 'codedeploy-agent':
    ensure => latest,
  }
 
  service { 'codedeploy-agent':
    ensure  => running,
    enable  => true,
    require => Package[ 'codedeploy-agent' ],
  }

Nothing is ever that easy of course. If I was using RedHat or AWSLinux I could just use the source attribute of the package type such as below to get around the lack of repository but I’m using Ubuntu.

  package { 'codedeploy-agent':
    ensure   => present,
    source   => "https://s3.amazonaws.com/aws-codedeploy-us-east-1/latest/codedeploy-agent.noarch.rpm",
    provider => rpm,
  }

So down the rabbit hole I go…

First, I needed a local repository which I setup via the puppet-reprepro module. Which worked well — except for the GPG part. What. A. Pain.

After that, I cracked the install script and fetched the .deb file to install…

$ aws s3 cp s3://aws-codedeploy-us-west-2/latest/VERSION . --region us-west-2
download: s3://aws-codedeploy-us-west-2/latest/VERSION to ./VERSION
$ cat VERSION
{"rpm":"releases/codedeploy-agent-1.0-1.751.noarch.rpm","deb":"releases/codedeploy-agent_1.0-1.751_all.deb"}
$ aws s3 cp s3://aws-codedeploy-us-west-2/releases/codedeploy-agent_1.0-1.751_all.deb . --region us-west-2
download: s3://aws-codedeploy-us-west-2/releases/codedeploy-agent_1.0-1.751_all.deb to ./codedeploy-agent_1.0-1.751_all.deb

…and dropped it into the directory the repo slurps files from.

Aaaannnnnd, nothing.

Turns out that the .deb AWS provides doesn’t provide an optional trait in its control file. But reprepro wants it to be mandatory. No problem.

$ mkdir contents
$ cd contents/
$ dpkg-deb -x ../codedeploy-agent_1.0-1.751_all.deb .
$ dpkg-deb -e ../codedeploy-agent_1.0-1.751_all.deb ./DEBIAN
$ grep Priority DEBIAN/control
$

Alright. Add in our line.

$ grep Priority DEBIAN/control
Priority: Optional
$

And now to package it all back up

$ dpkg-deb -b . ../codedeploy-agent_1.0-1.751_all.deb
dpkg-deb: building package 'codedeploy-agent' in '../codedeploy-agent_1.0-1.751_all.deb'.

Ta-da! The package is now able to be hosted by a local repository and installed through the standard package type.

But we’re not through yet. AWS wants to check daily to update the package. Sounds good ‘in theory’, but I want to control when packages are updated. Necessitating

  cron { 'codedeploy-agent-update':
    ensure  => absent
  }

Now we’re actually in control.

A few final comments;

  • It’d be nice if AWS would provide a repository to install their agents via apt — so I can selfishly stop managing a repo
  • It’d be nice if the Code Deploy agent had the Priority line in the control file — so I can selfishly stop hacking the .deb myself. The Inspector team’s package does…
  • It’d be nice if AWS didn’t install update scripts for their agents
  • The install script for Code Deploy and Inspector is remarkably different. The teams should talk to each other.
  • The naming convention of the packages for Code Deploy and Inspector are different. The teams should talk to each other.

(Whinging aside, I really do like Code Deploy. And Inspector looks pretty cool too.)

Saunter 2.0

Welp. After 2+ years of tinkering and appearing to be an absentee open source landlord, I just pushed Saunter 2.0.0 up to PyPI. When I list the things that have changed, it is rather silly to have allowed so much time to elapse, but…

  • Remove all references to Selenium Remote Control(RC). Enough time has passed that there are no excuses anymore to not be on WebDriver
  • Config file format has fundamentally changed to be YAML. This is why there has been a major version bump. The Saunter page has the details of the new format.

There are still some hiccups, but the main one is that random ordering of script execution and parallelization doesn’t work yet. I know how to fix it via monkey patch, but…

Per always, if you find any bugs, log them in github. I’ve fixed my notification settings to actually see these ones and will be going through the existing ones over the remainder of the month.

Stop Being A Language Snob: Debunking The ‘But Our Application Is Written In X’ Myth

The folks over at Sauce Labs just published a guest post on their blog I wrong on Stop Being A Language Snob: Debunking The ‘But Our Application Is Written In X’ Myth.

This doesn’t get debunked nearly enough. Consider this fair warning that this might end up being a 2015 theme.

Lessons learned from 19 months of a delivery manager

This is one of the talks I did at Øredev last week. As usual, my decks are generally useless without me in front of them. But lucky(?) for you, all the sessions were recorded.

CONFESSIONS OF A ROOKIE [DELIVERY] MANAGER from Øredev Conference on Vimeo.

But if you are too lazy to listen to me for 40 minutes, here is the deck and the content I was working from on stage. Of course, I don’t actually practice my talks so some content was added and others was removed at runtime, but…



WTF is a Delivery Manager?!?!

For about a year and a half I had to the title of ‘Delivery Manager’ which means a whole lot, and nothing at the same time. And therein lies it potency. Just as Andy Warhol famously said that ‘Art is anything you can get away with’, being a Delivery Manager is anything you make it. In my case it was essentially anything and everything to do with getting our application into the hands of the end users.

Tip: Don’t put yourself in a box

Before we landed on this title other ones we considered were ‘Doer of Stuff’, ‘Chaos Monkey’ (blantantly stolen from Netflix), and ‘Minister Without Portfolio.’ But we eventually went with the more business palatable of ‘Delivery Manager’. Since Delivery Manager is a made up title, it is useful to describe it in terms and titles people are used to seeing; Product Owner, Production Gatekeeper and Process Guardian are the three umbrella ones I most associated with. But even those could be sub-divided. And possibly sub-sub-divided. Its also important to recognize that the percentages of these roles are ever in flux. And just to keep things interesting, can sometimes be in conflict with each other.

Because of the mix of problems Delivery Managers will have to, erm, manage there is a certain skillset required to be effective at it. Or perhaps not a specific one, but a breadth of one. Testing, Development, Operations, Marketing, Systems, Accounting, etc.. And I would suggest that you have done a stint consulting as well. There is nothing like it in terms of being a crucible for problem identification and solving. That doesn’t mean of course that you have to be a perfect mix of all these things. It is inevitable that you will be more specialized in one over the other, and I would be suspicious of anyone who said they weren’t. I for instance come up through the testing ranks. Specifically the ‘context’ ranks. That, for me is my secret sauce.

And yes, there is a tonne of irony around the idea that I spent a decade saying ‘I am not a gatekeeper! I am a provider of information!’ to moving precisely into the gatekeeper role. But in that irony I learned a lot. Not just about being /a/ Delivery Manager, but about how /I/ am a Delivery Manager.

No*

While everything is important in one degree or another, this is perhaps the one thing I leaned on every single day. When faced with a request, the default answer is always No. Well, it is more ‘No* (* but help me to say Yes)’. And don’t be subtle or selective about the application of this rule. At 360 there is an entire department I dealt with on a daily basis and they could tell you my default answer is going to be ‘No’ to any request. But that doesn’t stop them from asking since they know about the asterisk. What it does is force them to think about their request ahead of time beyond simplistic ‘because’ terms.

This is not a new idea that I ‘discovered’. I blatantly stole it from someone who was at one point the Product Owner for Firefox (I think… I can’t find the article now, if you find it please let me know). It all boils down to an economics problem around opportunity cost. If you say Yes to everything then the queues will over flow and nothing will get done. But if you say No to everything and selectively grant Yeses then there is order [rather than chaos] in the pipes.

Tip: Learn about economics; specifically Opportunity Cost (but Sunk Costs are also useful to understand when involved in No* discussions)

Tip: Unless you really understand the problem you are being asked to solve, you cannot say yes

Mature organizations understand this at their core. It might be you that levels them up to this understanding though.

Frenemies

Being the person who always says No won’t always make you friends. At first at any rate. You will become everyone’s enemy … and everyone’s friend. Welcome to the balancing act. I would argue that if you are everyone’s friend all the time then you are not doing your job properly. Part of the animosity can be dealt with though explaining the asterisk, but also by communicating who ‘your’ client is. Remember the hats that are being warn have words like ‘Owner’, ‘Guardian’ and ‘Gatekeeper’. Your client in this role may not being whom it is people think it is. In fact, it almost assuredly isn’t. Yours is the application and the [delivery] pipeline.

Tip: The Delivery Pipeline is a product

This will cause friction; and depending on how your company is structured it could be a non trivial amount. But as long as you are consistent in your application of No* and are transparent in the reasonings why, in my experience, it is easily overcomable.

Tip: Do you know what business you are in? Is that the business the business thinks it is in? It’s really hard to win that battle.

Defence

The role of ‘Delivery Manager’ can sometimes be a lone wolf one, but at other times you will have people working for you [as I did]. It is critical to remember is that as a ‘people’ manager your primary goal is to protect everyone under you. Physically, psychologically and work-ly. You need to be able to do their job but also to let /them/ do it. Just because you /could/ be the hero doesn’t mean that it is healthy for you or them. Like you would a child, let them work through it and be ready to catch them if they start to fall. [The existence of that metaphor does not mean of course that you should treat them like kids though…] Don’t hold them to higher standards than you hold yourself to. But also don’t inflict yourself on them as well. I’m a workaholic (thanks Dad!); its unfair to put than onto others. I also don’t believe in work-life balance (especially in startups) favouring harmony instead — but what is harmonious for me is likely not the same for someone else.

In order to do that you need to constantly be running defence for your charges; human and software. Invite yourself to meetings, constantly be vigilant for conversations that will affect them. Which unfortunately means you miss out of plugging in your headphones and listening to music all day.

Tip: Ensure grief from No* comes back to you, not your people

Tip: People, not resources

Tip: Ask the people who work for you if they feel you have their back. If not, you’re doing something wrong.

You Will Screw Up

I tend not to speak in terms of absolutes, but here is a truth; You will screw up, potentially largely, in this role. You are making decisions that require a crazy amount of information to be assimilated quickly and if it is not perfectly done or you are missing any [maliciously or innocently] then you are hooped. And that’s ok. Pick yourself up, and go forward. That is the only way you can go. We no longer have the luxury of going back. Remember, tough calls are your job.

Bending to go forward is not a new thing. I’m sure I heard it a couple times before it really stuck, but I credit Brian Marick’s talk at Agile 2008 for that sticking. I can’t find a video of it [though didn’t try hard] but the text of it can be found a http://www.exampler.com/blog/2008/11/14/agile-development-practices-keynote-text.

Tip: Be careful though; screw up too much and Impostor Syndrome can set in. And it sucks. A lot. Get help. See Open Sourcing Mental Illness and Mental Health First Aid

Tip: Make sure your boss is onboard with the ‘go forward’ approach

Tip: Confidence is infectious, be patient zero

Know and be true to yourself
One of the biggest things I’ve learned in the last bit is around how /I/ function. Some people find the MBTI as hand-wavy and hokey, but I think its useful not in terms of how I choose to interact with people but in understanding how I am. I’m ENTP. Hilariously so. That’s not going to jive well with organizations that are ‘typed’ differently. That’s been a huge insight for me.

Tip: For a lark, take an MBTI test. Its heuristic, but still interesting

Being a geek I also think of things in terms of the classing AD&D alignment scale. I lean towards the Chaotic Good. We have a goal; there are no rules here. Especially ‘stupid’, ‘artificial’ ones.

And that has got me into trouble more than once. I don’t doubt that it will again in the future.

But I also have a strongly defined set of ethics and philosophy around how things should be done. Entrepreneurs don’t necessarily make good employees…

Putting a bow on it
Being a ‘Delivery Manager’ is great fun. Challenging as heck, but great fun and super rewarding. As someone who cares deeply about quality and the customer experience and has experience backed opinions on how to achieve them I don’t see myself going back to a ‘Just X’ role.

(P.S. I’m now available for hire if your organization needs a Delivery Manager)