Date: Mon, 02 Mar 2020 23:11:34 +0000
To: "~alpine/infra@lists.alpinelinux.org" <~alpine/infra@lists.alpinelinux.org>
From: remexre <nathan@remexre.xyz>
Reply-To: remexre <nathan@remexre.xyz>
Subject: Comcast blocks Alpine CDN
Message-ID: <IGOa-O0xRlQyJZlwUWhkziOQ2DR2Rusu5u6gQUdIcwM3C7S1CLgTrbGrwQE6MJA2nuQJCcdzjUAjQM_U7n87KIvkaS2ylNDEyYW1_vUheQY=@remexre.xyz>
Feedback-ID: 79Rapa0j15PPAPbHAOm8huJBsHT7ICMZgVQdrckxjbOiloEW9eSZkPmtcuEAo94pqR47CmVSrydx2xr2ujXNtg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Initial problem: curl http://dl-cdn.alpinelinux.org/ gets different
results inside our Comcast business-class network from anywhere else
(tried geographically near and far, on Comcast and other ISPs).

Everywhere else on the planet:

    $ dig +short dl-cdn.alpinelinux.org
    dualstack.global.prod.fastly.net.
    151.101.0.249
    151.101.64.249
    151.101.128.249
    151.101.192.249

Inside our network:

    $ dig +short dl-cdn.alpinelinux.org
    74.121.125.9
    74.121.125.8
    $ dig +short @1.1.1.1 dl-cdn.alpinelinux.org # !!!
    74.121.125.9
    74.121.125.8
    $ dig +short @8.8.8.8 dl-cdn.alpinelinux.org # !!!
    74.121.125.9
    74.121.125.8

DoH makes this a non-issue, so it's not a Cloudflare / DNS-based load
balancing thing:

    $ cloudflared proxy-dns --port 12345 &
    $ dig @127.0.0.1 +short -p 12345 dl-cdn.alpinelinux.org
    dualstack.global.prod.fastly.net.
    151.101.0.249
    151.101.64.249
    151.101.128.249
    151.101.192.249

Seems likely to be "Comcast SecurityEdge," which we got around the same
time as the symptoms were noticed. Per [0], Akamai's "SPS Secure
Business" [1] is (at least part of) what's powering it. This is backed
up by the whois results for the hijacking IPs:

    $ whois 74.121.125.8 | grep Organization
    Organization:   Akamai Technologies, Inc. (AKAMAI)

    $ whois 74.121.125.9 | grep Organization
    Organization:   Akamai Technologies, Inc. (AKAMAI)

curling that IP from offsite also gets 403's with the same content, and
Shodan results [2, 3] show that they're not (really) running HTTPS,
which (assuming competent configuration, I suppose) reinforces the idea
that this is intended for MITM'd traffic. Also of note: the ISP Shodan
assigns to those IPs is SKYE, which appears to be an "Intelligent DNS"
product from Nominum [4], which Akamai acquired [5]. BGP records back
this up [6].

[0]: http://archive.today/fASxW
[1]: http://archive.today/2aLcy
[2]: http://archive.today/ZMH6O
[3]: http://archive.today/1R18c
[4]: http://archive.today/gi21b
[5]: http://archive.today/hlH1S
[6]: http://archive.today/9rCOj