DevOps Fact Management | Ruby and INI

DevOps | Managing Eco System Facts

Fact management is at the heart of any good DevOps provisioning framework. The eco system (creation) framework is no different.

Facts are produced and consumed in well-defined ways that follow convention and raise productivity.

DevOps (SRE) facts can now be

  • produced via INI files, environment variables and code.
  • dynamically consumed through meta strings in INI file values
  • consumed by templates through auto evaluated placeholders
  • grouped as global, common or plugin-specific
  • re-evaluated (reproduced) to create another eco-system

# --
# --
# -- [eco.faculty.factbase.ini]
# --
# --   This fact (INI) file carries information for the eco-system platform.
# --   Refer to the managing facts article on build business websites.
# --
# ---  https://www.build-business-websites.co.uk/devops-facts-ruby-and-ini/
# --
# -- 
# -- -------------------------
# -- [[Stop Here]] Debug Tool
# -- -------------------------
# -- 
# -- Stop processing here to examine
# -- evaluation of the above facts.
# -- 
# -- stop.right.here = ruby>> exit
# -- 
# --


# --
# -- The next step is [double modularization] of the facts.
# -- 
# -- ----------------------------------------------
# -- [Include Statement] - 1st Fact Modularization
# -- ----------------------------------------------
# -- 
# -- First is the simple include statement. Best practice dictates that
# -- only the high level [.ini] can hold include statements. The restriction
# -- is designed to avoid long (including perhaps circular) fact file trees.
# -- 
# -- ----------------------------------------------
# -- [Fact Dependents] - 2nd Fact Modularization
# -- ----------------------------------------------
# -- 
# -- Fact dependency is the 2nd modularization effort.
# -- Examine each fact to understand what it depends on (is angled towards).
# -- This is the currently known list of fact dependents.
# -- 
# --   [1] - Time
# --   [2] - Plugin          (eg) src.pom.xml => plugin.dir => plugin.id
# --   [3] - Plugin and Time (eg) plugin.dir
# --   [4] - Host            (eg) ide.install.dir
# --   [5] - User
# --   [6] - Host and User   (eg) homde.dir
# --   [7] - Project         (eg) jar.projects
# --   [8] - Op System
# --   [9] - Environment     (eg) maven.version.str
# --  [10] - Command Line    (eg) --clean
# --  [11] - App Properties  (eg) application.properties group
# -- 
# --   Walk up fact dependents tree
# -- 
# --   Sometimes we must [WALK] up the fact dependents tree to the root in
# --   order to ascertain (from ancestor) the fact type.
# -- 


# --- -------------------------------------------------------------------- --- #
# --- [DEPENDENCY] => @db[:plugin][:dir]                                   --- #
# ---              => This dependency must be set before fact evaluation.  --- #
# --- -------------------------------------------------------------------- --- #

[aws]
iam.userid = ruby>> EnvVar.get_value "AWS_ACCESS_KEY_ID"
iam.secret = ruby>> EnvVar.get_value "AWS_SECRET_ACCESS_KEY"

[git]
       # --- ------------------------------------------------------------- --- #
       # --- [assume] command runs 1 folder level below working copy root. --- #
       # --- ------------------------------------------------------------- --- #
## IMPROVE BY USING THE MODULE PATH DIRECTIVE THEN DOUBLE FILE DIR NAME - MORE ROBUST - WE CANNOT GUARANTEE IN JENKINS SAY THAT PWD WILL BE HONORED.
local.dir = ruby>> File.dirname Dir.pwd
dot.git.dir = ruby>> File.join @db[:git][:local_dir], ".git"
revision = ruby>> GitFlow.wc_revision @db[:git][:dot_git_dir]
ssh.keyfile = ruby>> File.join Home.dir, "com.laundry4j.drive/library.ssh.access.keys/gitlab.laundry4j.private.key.pem"

# --
# -- ---------- = ----------------------------------------- #
# -- The fact l =  ayout format is [16 spaces] before the
# -- equals sig =  n then [2 spaces] after.
# --            =
# -- 6789012345 =  string value
# -- 6789012345 =  ruby>> 3 + 4
# -- ---------- = ----------------------------------------- #
# --

[plugin]
parent.dir.name =  eco.system.plugins
parent.dir.path =  ruby>> File.join @db[:git][:local_dir], @db[:plugin][:parent_dir_name]

month.span      =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yymo_mmm
month.span.b4   =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yymo_mmm_prev
month.stamp     =  ruby>> "eco." + @db[:plugin][:month_span]
month.stamp.b4  =  ruby>> "eco." + @db[:plugin][:month_span_b4]
month.dir       =  ruby>> File.join Home.dir, @db[:plugin][:month_stamp]
stamp.name      =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yyjjj_hhmm
stamp.hyph.name =  ruby>> (@db[:plugin][:stamp_name]).gsub ".", "-"
stamp.id        =  ruby>> @db[:plugin][:stamp_name] + Ch.p + @db[:git][:revision]
dir             =  ruby>> File.join @db[:plugin][:month_dir], @db[:plugin][:stamp_id]
src.dir         =  ruby>> File.join @db[:plugin][:parent_dir_path], @db[:plugin][:id]

[runtime]
parent.dir.name =  eco.system.plugins
parent.dir.path =  ruby>> File.join @db[:git][:local_dir], @db[:plugin][:parent_dir_name]

month.span      =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yymo_mmm
month.span.b4   =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yymo_mmm_prev
month.stamp     =  ruby>> "eco." + @db[:plugin][:month_span]
month.stamp.b4  =  ruby>> "eco." + @db[:plugin][:month_span_b4]
month.dir       =  ruby>> File.join Home.dir, @db[:plugin][:month_stamp]
stamp.name      =  ruby>> @db[:plugin][:id] + Ch.p + Stamp.yyjjj_hhmm
stamp.hyph.name =  ruby>> (@db[:plugin][:stamp_name]).gsub ".", "-"
stamp.id        =  ruby>> @db[:plugin][:stamp_name] + Ch.p + @db[:git][:revision]
dir             =  ruby>> File.join @db[:plugin][:month_dir], @db[:plugin][:stamp_id]
src.dir         =  ruby>> File.join @db[:plugin][:parent_dir_path], @db[:plugin][:id]

#--
#-- The [cmd.line.iptions] fact are the contents of the evoking
#-- command line held in an array.
#--
[cmd.line]
options         = ruby>> CmdLine.instance.args_cache


[s3]
upload.prefix   = s3put.
uploads.dir     = ruby>> File.join @db[:plugin][:dir], "s3_uploads"
month.bucket    = ruby>> @db[:plugin][:month_stamp]
month.bucket.b4 = ruby>> @db[:plugin][:month_stamp_b4]
monthwide.url   = ruby>> "s3://" + @db[:plugin][:month_stamp]

bucket.name     = ruby>> @db[:plugin][:month_stamp] # a [deprecated] fact

[gitlab]
backups.cmd     =  sudo gitlab-rake gitlab:backup:create
backups.dir     =  /var/opt/gitlab/backups
backups.cache   =  ruby>> @db[:plugin][:month_stamp]
backups.s3.dir  =  ruby>> "s3://" + @db[:plugin][:month_stamp]
ec2.acl.id      =  ruby>> "acl_" + @db[:plugin][:stamp_hyph_name]
ec2.acl.name    =  ruby>> "acl." + @db[:plugin][:stamp_id]
ec2.acl.desc    =  gitlab security group (acl) rules
ec2.name        =  ruby>> "ec2_" + @db[:plugin][:stamp_hyph_name]
ec2.tags.name   =  ruby>> @db[:plugin][:stamp_name] + ".key"
ec2.tags.group  =  ruby>> @db[:plugin][:stamp_name] + ".group"

[docker]
cmd.pre = ruby>> "sudo docker exec --user=" + Ch.dq + "mongodb" + Ch.dq + " -i container.mongodb bash -c " + Ch.dq
cmd.post = ruby>> Ch.dq

rm.images.yes = sudo docker rmi $(sudo docker images -q)
rm.images.no  = echo \"Note - this docker build may use cached images.\"
rm.images.cmd = ruby>> @db[:cmd_line][:options].include?( "--clean" ) \
                       ? @db[:docker][:rm_images_yes]               \
                       : @db[:docker][:rm_images_no]

[store]
admin.user.name     = ruby>> "admin.usr." + Stamp.yyjjj_hhmm
admin.user.password = ruby>> Secret.derive_alphanum
app.user.name       = ruby>> "app.usr." + Stamp.yyjjj_hhmm
app.user.password   = ruby>> Secret.derive_alphanum
database.name       = app_datastore
content.url         = http://192.168.0.14/content/database.git/
content.dir.path    = /var/lib/mongodb/content.library
data.zip.basename   = ruby>> "db.content." + Stamp.yyjjj_hhmm
data.zip.filename   = ruby>> @db[:store][:data_zip_basename] + ".zip"
data.filenames      = ruby>> GitFlow.file_names @db[:store][:content_url]

import.stmts = ruby>> MongoFlow.to_inport_stmts(       \
                       @db[:store][:data_filenames],   \
		       @db[:store][:content_dir_path], \
		       @db[:store][:database_name],    \
		       @db[:store][:app_user_name],    \
		       @db[:store][:app_user_password] \
		       )


[rest]
docs.url        = http://192.168.0.14/content/rest.documents.git/
docs.offset     = application.objects/
zip.basename    = ruby>> @db[:s3][:upload_prefix] + "application.objects"
zip.filename    = ruby>> @db[:rest][:zip_basename] + ".zip"


[host]
name.local      = ruby>> NetDns.instance.host_name
dns.name        = 192.168.0.14
username        = apollo
key.path        = ruby>> File.join Home.dir, "com.laundry4j.drive/library.ssh.access.keys/warehouse.iaas.private.key.pem"
rhost           = ruby>> HostBox.for_eco @db[:host][:dns_name], @db[:host][:username], @db[:host][:key_path]


[ide]
idea.iml.dir    = ruby>> File.join @db[:plugin][:dir], "idea_modules"
conf.repo.url   = http://192.168.0.14/content/intellij.conf.git/
conf.base.dir   = ruby>> File.join @db[:plugin][:dir], "ide_config"
conf.dir.name   = apollo.laundry4j
conf.home.dir   = ruby>> File.join @db[:ide][:conf_base_dir], @db[:ide][:conf_dir_name]

idea.prop.src   = ruby>> File.join @db[:plugin][:dir], "asset.idea.properties"
idea.prop.dir   = C:/Program Files (x86)/JetBrains/IntelliJ IDEA Community Edition 14.0.2/bin
idea.prop.dst   = ruby>> File.join @db[:ide][:idea_prop_dir], "idea.properties"

options.base.dir  = ruby>> File.join @db[:plugin][:dir], "ide_config/apollo.laundry4j/options"
default.xml.file  = project.default.xml
default.xml.path  = ruby>> File.join @db[:ide][:options_base_dir], @db[:ide][:default_xml_file]


[overwrite]
spec.filename   =  ruby>> @db[:plugin][:id] + ".overwrite.spec.json"
spec.filepath   =  ruby>> File.join @db[:plugin][:dir], @db[:overwrite][:spec_filename]


[s3.sync]
spec.filename   =  ruby>> @db[:plugin][:id] + ".s3.sync.spec.json"
spec.filepath   =  ruby>> File.join @db[:plugin][:dir], @db[:s3_sync][:spec_filename]


[maven]
jar.projects    = ruby>> %w[                                                \
	                    http://192.168.0.14/commons/laundry4j.facility  \
                            http://192.168.0.14/commons/laundry4j.mappable  \
		      	    http://192.168.0.14/commons/laundry4j.mappers   \
		            http://192.168.0.14/commons/laundry4j.clusters  \
		      	    http://192.168.0.14/football/footb4ll.net       \
		            http://192.168.0.14/commons/laundry4j.explorer  \
			   ]

# --
# -- The fact layout format is [16 spaces] before the
# -- equals sign and [2 spaces] after.
# --
# -- 6789012345 =  string value
# -- 6789012345 =  ruby>> 3 + 4
# --

install.dir     =  C:/Program Files/apache-maven-3.3.9
settings.xml    =  C:/Program Files/apache-maven-3.3.9/conf/settings.xml
version.dev     =  ruby>> Stamp.yy_jjj_hhmm + "-SNAPSHOT"
version.push    =  ruby>> Stamp.yy_jjj_hhmm

builds.dir      =  ruby>> File.join @db[:plugin][:dir], "maven_builds"
amalgam.dir     =  ruby>> File.join @db[:plugin][:dir], "maven_projects"
javadocs.dir    =  ruby>> File.join @db[:plugin][:dir], "maven_javadocs"
apidocs.dir     =  ruby>> File.join @db[:maven][:javadocs_dir], "site/apidocs"
pom.xml.src     =  ruby>> File.join @db[:plugin][:dir], "asset.amalgam.pom.xml"
pom.xml.dst     =  ruby>> File.join @db[:maven][:amalgam_dir], "pom.xml"

css.file.src    =  ruby>> File.join @db[:plugin][:dir], "asset.stylesheet.css"
css.file.dst    =  ruby>> File.join @db[:maven][:apidocs_dir], "stylesheet.css"

index.html.src  =  ruby>> File.join @db[:maven][:apidocs_dir], "overview-summary.html"
index.html.dst  =  ruby>> File.join @db[:maven][:apidocs_dir], "index.html"

cmd.prefix      =  ruby>> "mvn -f " + @db[:maven][:pom_xml_dst] + " clean "
cmd.javadocs    =  ruby>> @db[:maven][:cmd_prefix] + "javadoc:aggregate source:aggregate"

war.project     =  http://192.168.0.14/commons/laundry4j.web
war.module      =  ruby>> Pom.get_module_name @db[:maven][:war_project]
projects        =  ruby>> @db[:maven][:jar_projects].push @db[:maven][:war_project]
war.prj.dir     =  ruby>> File.join @db[:maven][:amalgam_dir], @db[:maven][:war_module]
war.pom.xml     =  ruby>> File.join @db[:maven][:war_prj_dir], "pom.xml"
war.run.cmd     =  ruby>> "mvn  -f " + @db[:maven][:war_pom_xml] + " cargo:run -P tomcat8x"

module.names    =  ruby>> Pom.get_module_names @db[:maven][:projects]
module.lines    =  ruby>> Refactor.sandwich_lines @db[:maven][:module_names], "<module>", "</module>", 16

no1.prj.dir     =  ruby>> File.join @db[:maven][:amalgam_dir], @db[:maven][:module_names].first
no1.pom.xml     =  ruby>> File.join @db[:maven][:no1_prj_dir], "pom.xml"
version.cmd     =  ruby>> "mvn -f " + @db[:maven][:no1_pom_xml] + " versions:set -DgroupId=com.* -DartifactId=* -DnewVersion=" + @db[:maven][:version_dev] + " -DgenerateBackupPoms=false"


############ mvn -f facility/pom.xml versions:set -DgroupId=com.* -DartifactId=* -DnewVersion=77.77.00-SNAPSHOT
###### http://localhost:8899/explorer-17.263.1858-SNAPSHOT

# --
# -- Create ruby/maven commands that act on one or more projects
# -- Command examples
# --   - system "mvn -f maven_projects/customer/pom.xml clean install"
# --
install.opts = clean install
install.cmd = ruby>> "mvn -f " + @db[:maven][:pom_xml_dst] + " clean install"

## cmd.prefix   = ruby>> "system " + Ch.dq + "mvn -f maven_projects/"
## cmd.postfix  = /pom.xml


[mongo.db]
props.src.path = ruby>> File.join Home.dir, "com.laundry4j.properties/application.properties"


[ci.stack]
docker.cmds.file = ruby>> File.join @db[:plugin][:dir], "ci.stack.build.cmds.ini"
####
#### Data becomes stale as the ci.stack.build.cmds.ini has PLACEHOLDERS
#### Reading now reads PLACEHOLDER laden content (not the placeholder values).
####
#### docker.cmds.array = ruby>> FactDb.to_values_array @db[:ci_stack][:build_cmds]
import.cmds.array = ruby>> Refactor.sandwich_array @db[:store][:import_stmts], @db[:docker][:cmd_pre], @db[:docker][:cmd_post]
#### rhost.cmds = ruby>> @db[:ci_stack][:docker_cmds_array] + @db[:ci_stack][:import_cmds_array]

props.jenkins = ruby>> File.join @db[:plugin][:dir], "volume.var.jenkins_home/com.laundry4j.properties"
props.winhome = ruby>> File.join Home.dir, "com.laundry4j.properties"
props.dst.dirs = ruby>> Array.new.push( @db[:ci_stack][:props_jenkins] ).push( @db[:ci_stack][:props_winhome] )
props.dst.name = application.properties

# --
# -- The fact layout format is [16 spaces] before the
# -- equals sign and [2 spaces] after.
# --
# -- 6789012345 =  string value
# -- 6789012345 =  ruby>> 3 + 4
# --

[javadocs]
s3sync.bucket   =  eco.javadocs.home
s3sync.folder   =  ruby>> @db[:maven][:apidocs_dir]

[pc.backup]
google.drive    =  C:/Users/apollo13/com.laundry4j.drive
drive.offset    =  ruby>> "google.drive." + @db[:host][:name_local]
build.assets    =  C:/Users/apollo13/asset-plate/app.football.archive.home

[gitlab.bkup]
rhost.cmds      =  ruby>> File.join @db[:plugin][:dir], "rhost.cmds.gitlab.bkup.ini"

Fact production and consumption begins when eco-system creation is kicked off. Fact consumption via template placeholders happens automatically at the framework level once each provisioning stage is completed. The eco framework recognises 3 provisioning stages – pre-provisioning, core provisioning and post provisioning.

Anti-Patterns | 7 Common Fact Mgt Problems

config file facts cannot be used lower down in the same file
– config fact values can only be static (not dynamic)
facts cannot travel config to code without touching the sides
– you are restricted to one programming language for fact production
config files can only be read once and cannot be (re-evaluated and) reused
– long lived passwords are encouraged (instead of short-lived tokens)
sensitive/secret facts have to be managed separately from the rest

Facts | From Config to Templates Without Touching Sides

4 out of 5 DevOps Code Statements | Matter of Fact


80% of IAAS (DevOps) code is fact acquisition, fact checking and error handling.
Fact acquisition statements do not change state.
You can remove 80% of your progam code.
Removing code massively raises productivity and reuse. It lowers bugs and time to market.
After removal the resulting 20% software are the juicy bits.
Juicy bits => are state changing and business specific code.

You can remove 4 out of 5 statements in your DevOps (CI/CD) software be it Bash, Python, Ruby, PowerShell, Perl or Go.

Fact acquisition code does not change state.

Pull out 80% of your IAAS DevOps code and you are left with the juicy bits which are state changing commands and business specific functionality.

7 Common Ways of Acquiring Facts

80% of your code is dedicated to fact acquisition. And you can throw a blanket over fact acquisition statements because most of them fall into one of just 7 categories.

In DevOps software, facts are acquired

  1. by string concatenation (union of other facts)
  2. from configuration files
  3. from local class/method command calls
  4. from lookups and regular expressions (lookdowns)
  5. from environment variables
  6. from derivations, amalgams and calculations
  7. from remote procedure calls

Stateless | Fact Acquisition Does Not Change State

Fact acquisition statements do not change state. Their acquisition is preparatory work before being

– used in command parameters
– replaced in file templates
– (potentially) returned to callers

fact meta-programming in ruby

Ruby code is easily written at runtime and used wisely, this feature makes it the ideal implementation system for DevOps fact management.

Whilst parsing the meta statements we can access facts embedded an eco-plugin key-value fact store called @facts.

Remember that meta-programming strays away from the confines and protections of compilers. Even though Ruby code is interpreted – it is important we don’t over-play the meta-programming card. If you can assign a value to a variable in one line (or with one call) – all is well and good. Any more than that and you should stay within the safety of standard software principles.

meta programming | ini file example

The aws username below is assigned to the string plain and simple.

[global]
aws.username = ABCDEFGHIJK
month.stamp = ruby>> Stamp.yy + Stamp.mo + "." + Stamp.mmm
s3.bucket = ruby>> ENV['USERNAME'] + "." + @facts[:global][:month_stamp]

However the next two lines prefixed with ruby>> denote ruby code that will be evaluated.
By way of example

evaluating (producing) month.stamp

The below INI file statement will evaluate the month.stamp fact within the [global] group.

month.stamp = ruby>> Stamp.yy + Stamp.mo + "." + Stamp.mmm

  • if Stamp.yy is 18 (for the year 2018)
  • and Stamp.mo is 04 (for April)
  • and Stamp.mmm is apr (for April)
  • then month.stamp will be 1804.apr

Now that the month.stamp fact is produced, it can be consumed (by convention) in three different ways.

  1. within ruby software with @facts[:global][:month_stamp]
  2. within another statement further on in the INI file (the same as ruby software)
  3. inside a template with @[global.month.stamp]

ini | consuming month.stamp to produce another fact

We can consume month.stamp to produce another fact within the same INI file – the newer fact must come after as the file is processed top to bottom.

s3.bucket = ruby>> ENV['USERNAME'] + "." + @facts[:global][:month_stamp]

The s3.bucket fact will be produced as apollo.1804.apr assuming the USERNAME environment variable is set as apollo.

Now we’ve seen how facts can be derived from within INI files. However, there is one more feature for frequent facts.

placeholders | injecting facts into templates

When creating eco-systems – we encounter a sea of templates (and commands), each with facts that must be injected at runtime.
If we knew the value of every fact before runtime – software would be nothing more than a sequence of command invocation statements.

Here are two placeholders in a format that will be automatically replaced by the eco DevOps software.

  • @[global.month.stamp] – is replaced by 1804.apr
  • @[commons.eco.id] – can be replaced by mongo.db

getting and setting facts in ruby software

When software in a provisioning stage reads, produces or derives a fact that others may need, it stashes that fact in the @commons registry.

fact aggregation | fact groups

Facts are grouped into silos for convenient management.

Facts in the ubiquitous application.properties file are held in the :app silo. They are consumed from application.properties (in the beginning) and eco plugins have the opportunity to add or overwrite property key/value pairs. At the end we backup the old application.properties file and stream the facts to produce a new one.

The commonly used fact aggregators are

  • :stamp – for eco reference who.what.when.where
  • :plugin – plugin source and target folders
  • :app – holding read in application.properties
  • :env – for the environment variables
  • :cmd – facts gleaned from the command line

As well as the above fact groups, each eco plugin will consume and produce its own facts accessible through a plugin id key.

sensitive | secret | facts

Facts containing one, either or both of the words secret and sensitive will be handled with care.

Their corresponding fact values will not be logged.

rich fact classes | not just strings

String values are all most fact managers can muster. That’s not rich enough for building DevOps eco-systems!

eco facts are not restricted to holding just string facts. We lean on Ruby’s ability to store any object of any class in arrays and maps. Hence eco fact values can be

  • an array of strings or integers or …
  • objects belonging to any class
  • hash maps of key-value pairs

why symbols | why underscores | why periods

Breaking with tradition goes against our better judgement. We have to honour

  • the tradition of ruby map access using symbols
  • the tradition of period separated template placeholders
  • the tradition of ruby variables containing underscores

The above 3 traditions make us write fact keys in slightly different ways depending on the context.

Leave a Reply

Your email address will not be published. Required fields are marked *