Here we go again.
Government blocked several social networks, totally shutdowned mobile network, caused by riots and meetings all around Kazakhstan. [1]
Messenger such as Telegram and WhatsApp is blocked. Mobile networks from Altel, Beeline, Tele2, Activ operators are not working, tried on all network types(LTE,3G,2G). Have reports from different regions, cities, and even from village. At least, mobile network shutdowned at 01:06(UTC+6) file:///tmp/1.jpg
bridges.torproject.org 10.293s error: Get "https://bridges.torproject.org": net/http: TLS handshake timeout
Люди говорили, что ничего не работает, а Instagram работает. Можете проверить?
Cloudflare Radar shows that the full shutdown happened after 10:30 UTC (16:30 local time) [2022-01-05]. But it was preceded by restrictions to mobile Internet access yesterday [2022-01-04].
The first disruptions reported affected mobile services, and we can see that at around 14:30 UTC yesterday, January 4, 2022, there was significantly less mobile devices traffic than the day before around the same time.
When we focus on other ASNs besides Kaz Telecom such as the leading mobile Internet services Tele2 or Kcell we can see a big drop in traffic yesterday [2022-01-04] after 16:00 UTC, confirming local reports. Mobile traffic did not drop to zero which may indicate throttling rather than a full shutdown. Today [2022-01-05], however, the Internet, mobile or not, is shut down.
Yesterday, January 5, 2022, after 18:00 UTC and for around three hours there was a return of some Internet services that happened at the same time Kazakh President Kassym-Jomart Tokayev announced in a televised speech that he appealed to a Russia-led security bloc to assist and “protect the state”. After 21:30 UTC the Internet shut down resumed.
@mushroom, do you know if normal DNS traffic (UDP port 53) is also shut down? I am thinking of the 2019 shutdown in Iran, where it was discovered that DNS was not blocked and that DNS tunnels might have worked to restore access.
If DNS is not blocked, then we can prioritize setting up some dnstt proxies.
@tango one guy wrote me from Kazakhstan right now. He said that DNS works and its possible to use it. But need to setup proxy servers
Okay. If you know someone who knows how to use a Unix command line, here is a test to see if dnstt will work. I have set up a dnstt-server that prints the current time when you connect to it. You may have to compile dnstt-client and give the user a binary, if they are not able to download the source code. In one terminal, run dnstt-client:
dnstt-client -udp tns.rinsed-tinsel.site:53 -pubkey 6f78064ecc2147e8f5de5c565e4ad1e6aa28f866b2d28c3685ceca2697a37470 t.rinsed-tinsel.site 127.0.0.1:7000
In another terminal, connect to the client side of the tunnel:
nc -v 127.0.0.1 7000
The dnstt-client terminal should show that a stream began and ended:
2022/01/08 17:25:28 begin stream XXXXXXXX:3
2022/01/08 17:25:29 end stream XXXXXXXX:3
The other terminal should show the current time from the server:
Sat 08 Jan 2022 05:25:28 PM UTC
If it does not work, try again, this time sending queries recursively through the ISP resolver (i.e., nameserver from /etc/resolv.conf) instead of connecting to the dnstt-server directly.
dnstt-client -udp <ISP_DNS_RESOLVER_IP>:53 -pubkey 6f78064ecc2147e8f5de5c565e4ad1e6aa28f866b2d28c3685ceca2697a37470 t.rinsed-tinsel.site 127.0.0.1:7000
You have to use -udp mode. -doh and -dot mode are not likely to work during a shutdown. Unfortunately, -udp mode is easy to detect and block, if the censor knows what to look for. But the contents of the tunnel will still be encrypted.
If the manual test works, the easiest immediate solution to get access is probably to use one of the third-party Android VPN apps that has dnstt capability. I am not involved with any of these, and I don’t know whether they are actually trustworthy or safe. This is not an endorsement. I think they make you watch and advertisement before you get access. But they will be good enough for a test, and perhaps to bootstrap a more stable connection.
Here are a few apps I know of. You can find these in the Play Store with a search for “dnstt” or in some cases “slowdns”. With all of these, you have to download a primary app, plus a secondary dnstt plugin app. If the user cannot access the Play Store, it should be possible to download the APK files and send them through another channel.
- TLS Tunnel + TLS Tunnel DNSTT Plugin
- One VPN + One VPN - DNSTT Plugin
- PurpleVPN + PurpleVPN - DNSTT Plugin
- HkH VPN + HKH VPN - Plugin
You can find some video tutorials for these on YouTube, for example:
If these tests work, a next step is probably to talk to Access Now about establishing proxy servers. You can set up dnstt as a SOCKS proxy, a Tor bridge, a Shadowsocks plugin, and in other ways.
From IODA, it looks like access has been partially restored a few times, for a few hours each:
You can see the same pattern in AS9198 (KAZTELECOM-AS). But, for example, AS8200 (UPLINK-AS) only shows the 2022-01-08 interval of connectivity, not the 3 earlier ones.
That guy already tunnel traffic via dnstt to his own server. It works. Speed isn’t high, but possible to write text massages.
It should be possible to run ICMP tunnel or something. I’m pretty sure if DNS is working, than not all protocols with direct connectivity are blocked.
ICMP не работает. Я пробовал снаружи пингануть пару адресов, но ни один не ответил. Изнутри ICMP идёт только к 8.8.8.8. TCP, UDP не работает, за исключением dns на 53 к операторскому и гугловскому резолверам. Вообще, у каждого провайдера свой тип блокировки. Сейчас пишу про Билайн, но говорят, что на Казахтелекоме намного проще, там можно просто https проксей. Но, ещё раз говорю, в каждом регионе и у каждого оператора свои заморочки.
SOCKS5 proxy 3785 port works fine. Not sure why, VoIP using skype and other services works as well, so I guess 3785 may be used for VoIP
in general, it’s easy to configure in telegram, but if clients are able to configure proxy on their OS(for example using proxifyer) https and all other traffic works as well.
This has been tested in at least 3 regions.
That’s great, thank you for the information.
I am not familiar with that one either. nmap-services calls it bfd-echo “BFD Echo Protocol”. RFC 5881 says it is a UDP protocol:
BFD Echo packets MUST be transmitted in UDP packets with destination UDP port 3785 in an IPv4 or IPv6 packet.
comment is saying that proxy only works on main provider in Kazakhstan - KazakhTelecom.
Yeah, if it’s not VoIP I have no idea why it works. I guess people found it out by brute-forcing different ports
Also, working VoIP makes me think that there are other ports open on this provider as it also provides landline in Kazakhstan
Here is an obfs4 bridge on port 3785 (IPv4 and IPv6) to try in Tor Browser:
Bridge obfs4 172.105.56.235:3785 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:3785 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
Documentation for entering bridges:
If this works, we may be able to set up more, for as long as it lasts.
Hello, I am that guy from Kazakhstan. Everything is as zhenyolka says. (Beeline)
The IPv4 obfs4 bridge is working!
I did some port scans. It looks like some other ports to try are 179, 646, 3784, 3785, 4784, 5060.
First I did a scan to see if any hosts in the /24 neighborhood of gov.kz were reachable on port 3785. Only one of them was, 195.12.114.89 (whois), which is part of “National Information Technologies Joint-Stock Company”:
# nmap -PS3785 -sn -n gov.kz/24
Nmap scan report for 195.12.114.89
Host is up (0.21s latency).
Nmap done: 256 IP addresses (1 host up) scanned in 15.57 seconds
Then, I scanned all the ports on that host. 6 ports were responsive, including 3785:
# nmap -n -PS3785 -p- --reason 195.12.114.89
Nmap scan report for 195.12.114.89
Host is up, received reset ttl 236 (0.21s latency).
Not shown: 65529 filtered ports
Reason: 65529 no-responses
PORT STATE SERVICE REASON
179/tcp closed bgp reset ttl 233
646/tcp closed ldp reset ttl 236
3784/tcp closed bfd-control reset ttl 234
3785/tcp closed bfd-echo reset ttl 234
4784/tcp closed bfd-multi-ctl reset ttl 233
5060/tcp open sip syn-ack ttl 50
Nmap done: 1 IP address (1 host up) scanned in 344.21 seconds
A port scan could also be a way to discover what foreign ports are accessible from inside Kazakhstan. You need to target a host that responds to every port (with either a SYN/ACK or a RST), like scanme.nmap.org. Any port that has reason syn-ack or rst is making it through the shutdown. Any port that has no-response is blocked by the shutdown.
# nmap -v -n -Pn -p- -T4 --reason scanme.nmap.org
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up, received user-set (0.23s latency).
Not shown: 65531 closed ports
Reason: 65531 resets
PORT STATE SERVICE REASON
22/tcp open ssh syn-ack ttl 55
80/tcp open http syn-ack ttl 55
9929/tcp open nping-echo syn-ack ttl 56
31337/tcp open Elite syn-ack ttl 56
Nmap done: 1 IP address (1 host up) scanned in 108.98 seconds
# nmap -v -n -Pn -p- -T4 --reason -6 scanme.nmap.org
Nmap scan report for scanme.nmap.org (2600:3c01::f03c:91ff:fe18:bb2f)
Host is up, received user-set (0.23s latency).
Not shown: 65532 closed ports
Reason: 65532 resets
PORT STATE SERVICE REASON
22/tcp open ssh syn-ack ttl 55
80/tcp open http syn-ack ttl 56
31337/tcp open Elite syn-ack ttl 56
Nmap done: 1 IP address (1 host up) scanned in 146.68 seconds
I see you have already set up the bridge. But Softether VPN also allows to encapsulate VPN in DNS or ICMP. I don’t know if this is available for public VPNGate servers.
Провайдер Казахтелеком.
Интернет отключили 17:00 05.01.2022
Дальше отключили полностью мобильную связь, не ловило в любых режимах(2G, 3G, 4G)
Через несколько дней включили мобильную связь, но звонки до сих пор отвратно работают.
Вывод traceroute:
traceroute to dns.google (8.8.4.4), 30 hops max, 60 byte packets
1 _gateway (192.168.100.1) 1.340 ms 2.627 ms 2.562 ms
2 82.200.242.218 (82.200.242.218) 6.005 ms 6.513 ms 7.061 ms
Дальше одни звездочки
С этим выводом я воодушёвленный пошёл проверять связь с другими клиентами сети казахтелекома. И пинг был(3 хопа)! И даже больше, кажется на них нету фильтра.
Мы спокойно прокидывали порты, HTTP, SSH, и прочие протоколы.
До других IP происходит полная фильтрация(даже icmp). Режим белый список.
В белом списке находится:
dns.google(8.8.8.8), akorda.kz, IP банков и государственых новостных агенств, а также мобильных операторов
Ставлю предположение, что фильтрующее обуродование на третьем/четвертом хопе стоит.
С этим уже кажется можно получить доступ в интернет, через dns туннель. Но к сожалению у меня нету сервера за рубежом. Также скорее фильтрация, крайне сильная с урезанием функционала до минимума, так я не смог icmp трафик сделать до всех хостов в whitelist. Кроме altel.kz
09.01.2022 дали доступ ко всем подсетям hoster.kz, neolabs.kz, ps.kz. Мне кажется или у хостингов есть интернет, так как судя по зеркалу репозиториев там они относительно свежие.
Сегодня, 10.01.2022 в 8:45 дали интернет.
В Астане давали интернет уже 3 дня назад. Но временно, с 8:00-13:00
I verified that shadowsocks+v2ray works just fine trough 3785 port.
Almaty, Kazakhtelecom
OpenVPN on port 3785 (udp) works.
Some information:
$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time ms
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=100 time= ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=100 time= ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=100 time= ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time ms
$ dig google.com @8.8.8.8
; <<>> DiG <<>> google.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 272 IN A 173.194.222.113
google.com. 272 IN A 173.194.222.138
google.com. 272 IN A 173.194.222.100
google.com. 272 IN A 173.194.222.102
google.com. 272 IN A 173.194.222.101
google.com. 272 IN A 173.194.222.139
;; Query time: msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jan 10 18:58:53 +06 2022
;; MSG SIZE rcvd: 135
$ curl https://8.8.8.8
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://dns.google/">here</A>.
</BODY></HTML>
hoster.kz, neolabs.kz, ps.kz - timeout
altel.kz, akorda.kz - works
(If you want to investigate, you can contact me using Discord (invite: rTjTadmYvt))
TCP, UDP, ICMP трейсы (-T, -U, -I) до 8.8.8.8 нормально выглядят в Казахтелекоме?
Wrote you PM but it seems that Kazakhstan net is getting shut down again.
#Internet connectivity was shutdown in #Kazakhstan again at ~1300 UTC after 6th brief service restoration since shutdowns started on Jan. 5. @cloudflareradar shows that this one saw peak traffic 2x or more as compared to previous restorations.
That matches the IODA signals as well. The restoration of access of January 10 (starting 00:00 UTC) lasted 13 hours and seemed to include more networks than past ones.
We’ve switched all Lantern (https://lantern.io) servers in the region to listen on 3785, 5060, as well as randomized high ports.
I found that port 179 works fine on both ISPs (KazakhTelecom and Beeline).
Thanks @sasha0552 for help!
The Tor community team posted a guide on how to get working bridges. You will not be able to use BridgeDB or Moat; instead, email frontdesk@torproject.org with subject “bridge kz”.
Thank you for the information. I opened port 179 on the the bridge from earlier as a backup in case 3785 gets blocked.
Bridge obfs4 172.105.56.235:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0
I did it with port forwarding:
iptables -A PREROUTING -t nat -p tcp --dport 179 -j REDIRECT --to-ports 3785
Репортирую, что вчера при включение интернета было ограничение скорости до 3Мбит /с примерно. На https видимо максимум по 20Кбит/с, не смог даже обновить репозитории.
Thanks! Today we heard from a user that Beeline is blocking 3785:
К сожалению, у Билайн Казахстан заблокирован порт 3785, есть ли другой способ обхода блокировки?
We will try your bridge.
Фиксирую падение интернета до начала шатдауна:
ISP: Beeline KZ (“Интернет Дома”).
Время: 17:10 - 17:24 (GMT+6).
Таймаут до Google DNS.
Внутренние DNS провайдера остались доступны - dnstt работал.
Порты tcp/179 и tcp/3785 были заблокированы.
Tor used to have fteproxy bridge, which claimed to masquerade as unencrypted http. Although, I think it would be easy to block by fingerprint. Binary is still available, but no one is providing this type of bridge right now. However I would like to test it.
Binary is static with python embedded inside.
In networks with low bandwidth it would be useful to use HandyCache caching proxy.
It can also decrypt and cache https traffic, but this functionality in the trial mode only works for the first 30 minutes after each start of application (and then you need to restart HC). There is an English and Russian interface. Works in Wine.
IODA measurements say that access has been restored since about 2022-01-11 00:00 (06:00 Almaty time). Does that match people’s experience? I can access gov.kz now.
Yes, there were no more shutdowns.
I have shut down this bridge now.
Here are graphs of its usage over the past few days:
https://metrics.torproject.org/rs.html#details/0E9783A73F029E0910FD72F1EC120CA818868DA0
@anadahz pointed me to a RIPE Labs blog post on the shutdown. It notes that despite being “shut down,” networks in Kazakhstan were still present in the global BGP routing tables, which matches our experience with certain ports being unblocked. It also has some analysis of different levels of access in e.g. data centers versus residential connections.
It is difficult to pinpoint the cause of the outage. However, the affected networks have remained visible in the global routing system (BGP), which means they’ve remained “connected” to the Internet even though they have not been able to send or receive packets. The timing of the outage was synchronised, suggesting it was the result of some centralised action, although we do see small variations per region.
If we try to distinguish between RIPE Atlas vantage points in infrastructure - i.e. RIPE Atlas anchors and other probes with tags that suggest they are in data centres - we see differences in how connectivity developed over the last few days.
The figure below shows infrastructure vantage points in red. While connectivity for most of these vantage points went down in the last few days, it looks like most are able to send and receive packets to/from the Internet again since around midnight UTC on Friday 7 January. The other vantage points, which we think are mostly near end-users show that over the last few days there were periods of multiple hours where some of these vantage points had Internet connectivity.
After a few hours where almost all of our RIPE Atlas vantage points were online again, we see a drop again. If we look at infrastructure (data centres) versus other probes we do see that roughly half of the other probes (homes, offices, etc.) go down again, but many stay connected.
The comments on the post link to an interactive notebook for analyzing outages using RIPE Atlas.
I want to make a post that summarizes the important lessons from the January 2022 shutdown in Kazakhstan. I have written a draft in English (about 1200); is anyone willing to translate it to Kazakh and Russian before I post it?
https://pad.riseup.net/p/RQ1I5QI01qRfWZJBrUBD
You can also edit the document to add something you think is important. I’m planning to make the post next Monday, 2022-01-07.
I made translation to Russian. This is not a direct word-by-word translation, but more like my interpretation of the text according to typical Russian text constructs.
Thank you, I appreciate it. I think that is the right way to translate.
Posted.
An article by Katia Patin gives some details about how working ports were discovered, and efforts to establish proxies.
2022-01-27
Kazakhstan shut down its internet. These programmers opened a backdoor (archive)
Обойти национальный шатдаун: как молодые IT-специалисты вернули интернет тысячам казахстанцев (archive)
A senior software engineer at LinkedIn in Toronto, Maksat Kadyrov jumped into action when he lost touch with his brother in Almaty. He went live on Instagram, looking to crowdsource a way to reach his family. … He live streamed on Instagram for hours as they scanned some of the more than 65,000 existing ports. During the live stream, they found five open ports, tested them and were able to establish a connection. They later learned that it was a bug in outdated Cisco equipment, used widely by Kazakh telecom operators, which had accidentally kept these ports open. Kadyrov, Maksut and the others used these open ports to support their operation, crowdsourcing funds or footing the cloud computing bill themselves from service providers like Digital Ocean and Amazon.
Over the next few days, the loosely organized group set up dozens of proxy servers — first for Telegram and later even for internet browsers like Firefox. Maksut admits their user estimates aren’t exact; not all of them had a chance to collect data. But more recently, on January 19, Zharaskhan Aman, a software engineer at Facebook in London, rounded up some of the numbers he had from Telegram analytics showing that the 9 servers he raised alone had 155,762 users from Kazakhstan between January 4 and 11. … Based on user traffic provided by Telegram, Maksut estimates the group got between 300,000 to 500,000 people online on the message app during the five-day shutdown. … Sharing connection instructions by Telegram, email and text, members of the group said they were overwhelmed with demand. Within 24 hours Kadyrov said he had more than 2,000 requests for access to his servers, which he had been sending out one-by-one.
Когда старший инженер-программист LinkedIn в Торонто Максат Кадыров потерял связь со своим братом в Алматы, он решил, что пора действовать. … В течение нескольких часов он вел прямую трансляцию в Instagram, пока они сканировали несколько из более чем 65 тысяч существующих портов. Во время прямого эфира они обнаружили пять открытых портов, протестировали их и смогли установить соединение. Позже они узнали, что некоторые порты оказались случайно открыты из-за ошибки в устаревшем оборудовании Cisco, широко используемом казахстанскими операторами связи. Кадыров, Максут и другие использовали эти открытые порты для поддержки своей операции и покупали серверы у Digital Ocean, Amazon и других провайдеров на деньги, собранные краудсорсингом или свои собственные средства.
В течение следующих нескольких дней группа энтузиастов установила десятки прокси-серверов — сначала в Telegram, а затем даже в интернет-браузерах, таких как Firefox. Максут признает, что его оценка количества пользователей не точна — не у всех была возможность собрать данные. Но 19 января Жарасхан Аман, инженер-программист, работающий в Facebook в Лондоне, изучил аналитику Telegram и посчитал, что только 9 поднятыми им серверами с 4 по 11 января воспользовались 155 762 пользователя из Казахстана. … По оценкам Максута, основанным на данных о посещаемости Telegram, за время пятидневного отключения группа дала доступ к приложению от 300 до 500 тысячам человек. … Обмениваясь инструкциями по подключению через Telegram, электронную почту и СМС, члены группы говорили, что они не справлялись с потоком запросов. Кадыров говорит, что всего за 24 часа ему поступило более 2 тысяч запросов на доступ к его серверу, которые он рассылал по одному.
I found Katia’s article as a reference in the paper Government Internet Shutdowns Are Changing. How Should Citizens and Democracies Respond? (2022-03-31).
Интересно, как при такой тотальной блокировке проще всего проверить все порты? Может быть есть готовые сервисы которые отвечают на любом порте?
There are some notes about port scanning higher in the thread and in another post.
About hosts that respond on any port, scanme.nmap.org is one such, but there are many. The host does not have to have every port open (SYN/ACK response); it is enough if it exposes its closed ports (RST response).