This is a tool to visualise the web of dependencies present in a large configuration of ThoughtWorks’ Go system for continuous delivery.
Skip to an example built from our pipeline definitions.
Circles represent pipelines
squares represent pipeline templates
triangles represent source code repositories.
Arrows represent dependencies and the size of each shape grows with more incoming dependencies. Pipelines in the same groups are coloured similarly.
It uses the excellent work of Mike Bostock on D3.js, adapted from a number of his examples (e.g. http://bl.ocks.org/mbostock/4062045).
It works by fetching the configuration XML over HTTP, either from the Configuration API (provided you are a logged in Go Admin) or from a supplied URL. It then extracts the declared dependencies into JSON and passes this to D3.js to be displayed as SVG.
Getting Started
Use the hosted HTML page to explore your dependencies. It can build the graph in three ways: fetching and parsing the XML, using graph data encoded into the URL or from a generated JSON file hosted remotely. Here are instructions (with the easiest first):
Paste in your XML
If you scroll down there’s a text box you can paste your XML in to. Click load, and your graph will be built from the source. You can't save the JSON at the moment, or share the chart generated (watch this space).
Sharing graphs as URLs
Once you've built the graph, a link will appear at the top right which contains the JSON metadata compressed and encoded. For large graphs the compressed data may be too large for some browsers to show, so I've set a hard limit of 10k characters (the limit of Chrome's omnibox) beyond which you’ll see message suggesting you try hiding labels. With the labels hidden the link it generates will not contain the labels, so is safe to share with the public.Fetching XML from Go
Simply point to the configuration XML by supplying a configUrl parameter like this:
http://localhost:8081/pipeline-dependencies.html?configUrl=http://yourgoserver/go/admin/config_xml
Since it makes cross origin requests, you'll need to make sure the target server (Go / JSON location) authorises this page as an origin, using the Access-Control-Allow-Origin: http://mrmanc.github.io header.
You’ll also need to issue the Access-Control-Allow-Credentials: true header from your Go instance to ensure your browser is allowed to send your cookie credentials to Go (as fetching config is privilidged). Read more about Cross Origin Resource Sharing
Loading graphs from JSON files
If you have problems with long URLs you may wish to load the graph data from JSON. Currently you’ll need to do this by hosting the JSON file somewhere and telling the page where to find it, by supplying it’s address as a parameter called ‘u’ like this: http://mrmanc.github.io/go-dependency-force-layout/pipeline-dependencies.html?u=http://mrmanc.github.io/go-dependency-force-layout/examples/27012014.json.
Your JSON location will need to issue the Access-Control-Allow-Origin: http://mrmanc.github.io header. At some point I will make it possible to save the JSON generated by the page, and load graphs without hosting JSON files remotely.
Hosting your own HTML
If you want to clone the project and use the page offline, or you don’t want to fiddle with cross origin headers, you’ll need to start a local web server. Using a file:// url will not work. A shell script is included to quickly get a server running using Python: see start-http-server.sh.
The tool will default to the included sample.xml if you do not supply the parameter, to give you an idea of how it works.
Why do this?
I built this to see if it could help me understand how our organisation was using Go, and it proved very enlightening.
I also found it interesting to look at previous historical versions of the configuration, which you can get at using the Go configuration API provided you know the MD5 hash. If you don't, you can use git to query the configuration git repository on the server itself, using something like this:
git --git-dir=/data/go/db/config.git/.git/ log | tail -100
You can only go back as far as your upgrade to v2.2 though as that is when version control was brought in.
If, like me, yours does not fit on the screen, hit zoom out a few times and refresh the page.
Here is a link to an example built from our pipeline definitions.
What's next?
⊠ Specify XML location in parameter
⊠ Add code to allow Cross Origin Resource Sharing
⊠ Add arrowheads to indicate direction of dependency
⊠ ‘Compiled’ graphs with data in the URL (like Web Sequence Diagrams) to allow you to share without leaking config
⊠ Compress compiled dependency data to shorten the URL
⊠ Hiding of labels for sharing
⊠ Provide obfuscated realistic example to play with
⊠ Use GitHub Pages to host the HTML file
⊠ Allow user to generate a graph based on pasted in XML
◻ Allow user to copy / save generated JSON to share with others
◻ Allow user to open generated JSON pasted in
◻ Replace ugly URL parsing with a decent library
◻ Fix bug where you can ‘show’ labels on a shared chart with no labels
◻ Links to pipelines from nodes
◻ Bendy connectors like this D3.js example
◻ Test Cross Origin Resource Sharing on a real Go server
◻ Improve my butt ugly procedural JavaScript
◻ Error messages when the graph fails to load
◻ Super secret cool next idea