much-data/kwork-api
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove mitmproxy (not practical for cross-network phone capture)

Browse Source
This commit is contained in:
Claw 2026-03-23 01:16:03 +00:00
parent 27ff64e02e
commit c0935e2c19
6 changed files with 0 additions and 667 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

312
mitmproxy/INSTALL.md
View File

@ -1,312 +0,0 @@
# Установка и запуск mitmproxy на VPS
## 1. Установка mitmproxy
### Вариант A: pip (рекомендуется)
```bash
# Установить pip
apt-get update && apt-get install -y python3-pip
# Установить mitmproxy
pip3 install mitmproxy
# Проверка
mitmdump --version
```
### Вариант B: Docker (если pip не работает)
```bash
# Установить Docker
apt-get install -y docker.io
# Запустить в контейнере
docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy \
mitmdump -p 8080 --mode regular
```
### Вариант C: apt (старая версия)
```bash
apt-get install -y mitmproxy
```
---
## 2. Настройка firewall
```bash
# Открыть порт 8080
ufw allow 8080/tcp
# Проверить
ufw status
```
---
## 3. Генерация сертификата
```bash
# Первый запуск создаст сертификат
mitmproxy --version
# Сертификаты будут в ~/.mitmproxy/
ls -la ~/.mitmproxy/
```
---
## 4. Запуск с авторизацией
### Быстрый старт
```bash
cd /root/kwork-parser/mitmproxy
chmod +x start.sh
./start.sh
```
### Вручную
```bash
mitmdump \
-s /root/kwork-parser/mitmproxy/auth.py \
--set auth_user=claw \
--set auth_pass="KworkParser2026!" \
-p 8080 \
--mode regular \
-w /root/kwork-parser/mitmproxy/flows.mitm
```
### Как systemd сервис
```bash
# Установить сервис
cp /root/kwork-parser/mitmproxy/mitmproxy.service /etc/systemd/system/
# Запустить
systemctl daemon-reload
systemctl enable mitmproxy
systemctl start mitmproxy
# Проверить статус
systemctl status mitmproxy
# Логи
journalctl -u mitmproxy -f
```
---
## 5. Настройка телефона
### Android
1. Настройки → Wi-Fi
2. Долгое нажатие на сеть → Modify network
3. Advanced → Proxy: Manual
4. Proxy hostname: `<VPS_IP>`
5. Proxy port: `8080`
6. Сохранить
### iPhone
1. Настройки → Wi-Fi
2. Нажать на ⓘ рядом с сетью
3. Configure Proxy: Manual
4. Server: `<VPS_IP>`
5. Port: `8080`
6. Authentication: ON
- Username: `claw`
- Password: `KworkParser2026!`
7. Save
### Установка сертификата (Android)
1. Открыть браузер на телефоне
2. Перейти: `http://<VPS_IP>:8080/mitmproxy/mitmproxy-ca.cer`
3. Скачать сертификат
4. Настройки → Security → Install certificate
5. Выбрать файл: `mitmproxy-ca.cer`
6. Name: `mitmproxy`
7. OK
### Установка сертификата (iPhone)
1. Safari: `http://<VPS_IP>:8080/mitmproxy/mitmproxy-ca.cer`
2. Allow → Profile downloaded
3. Настройки → Profile downloaded → Install
4. Настройки → General → About → Certificate Trust Settings
5. Включить доверие для mitmproxy
---
## 6. Проверка работы
### С телефона
```bash
# Открыть браузер на телефоне
# Перейти на http://example.com
```
### На VPS
```bash
# Посмотреть логи
tail -f /root/kwork-parser/mitmproxy/mitmproxy.log
# Или через скрипт
python3 /root/kwork-parser/scripts/view_flows.py --last 10
```
---
## 7. Просмотр логов
### Последние запросы
```bash
python3 /root/kwork-parser/scripts/view_flows.py --last 20
```
### Только Kwork
```bash
python3 /root/kwork-parser/scripts/view_flows.py --domain kwork.ru --last 50
```
### Только JSON
```bash
python3 /root/kwork-parser/scripts/view_flows.py --json --last 20
```
### Полная информация
```bash
python3 /root/kwork-parser/scripts/view_flows.py --domain kwork.ru --full
```
### В реальном времени
```bash
tail -f /root/kwork-parser/mitmproxy/flows.mitm | \
python3 -c "
import sys
from mitmproxy import io
for flow in io.FlowReader(sys.stdin.buffer).read():
if hasattr(flow, 'request'):
print(f'{flow.request.method} {flow.request.url}')
"
```
---
## 8. Безопасность
### Что защищено
✅ **Авторизация** — Basic Auth (логин/пароль)
✅ **Firewall** — только порт 8080 открыт
✅ **Изоляция** — mitmproxy работает от root (для systemd)
### Рекомендации
1. **Смени пароль** после первого запуска:
```bash
# В auth.py и start.sh
auth_pass="НовыйПароль2026!"
```
2. **Ограничь IP** (если возможно):
```bash
# В start.sh добавить:
--set allow_hosts=["твой_домашний_IP"]
```
3. **Используй HTTPS** для mitmweb:
```bash
mitmweb --web-port 8081 --ssl-keyfile key.pem --ssl-certfile cert.pem
```
4. **Очищай логи** регулярно:
```bash
# Cron: очищать раз в день
0 0 * * * truncate -s 0 /root/kwork-parser/mitmproxy/flows.mitm
```
5. **Не логируй чувствительные данные**:
```bash
# Игнорировать banking сайты
mitmdump --set 'ignore_hosts=~u (bank|payment|stripe)"'
```
---
## 9. Troubleshooting
### Ошибка: Address already in use
```bash
# Проверить кто использует порт
lsof -i :8080
# Убить процесс
kill -9 <PID>
```
### Ошибка: Certificate not trusted
```bash
# Переустановить сертификат на телефоне
# Удалить старый из доверенных
# Скачать заново: http://<VPS_IP>:8080/mitmproxy/mitmproxy-ca.cer
```
### Ошибка: Authentication failed
```bash
# Проверить логин/пароль
# Логин: claw
# Пароль: KworkParser2026!
# В настройках прокси телефона включить Authentication
```
### mitmproxy не стартует
```bash
# Проверить логи
journalctl -u mitmproxy -n 50
# Проверить конфиг
cat /root/kwork-parser/mitmproxy/mitmproxy.log
```
---
## 10. Команды для Claw (AI)
```bash
# Посмотреть последние запросы
python3 /root/kwork-parser/scripts/view_flows.py --last 20
# Найти API эндпоинты
python3 /root/kwork-parser/scripts/view_flows.py --json --domain kwork.ru
# Экспортировать в HAR
python3 /root/kwork-parser/scripts/mitmproxy-to-har.py \
/root/kwork-parser/mitmproxy/flows.mitm \
/root/kwork-parser/mitmproxy/export.har
# Анализ HAR
jq '[.log.entries[] | select(.request.url | contains("kwork"))] | length' \
/root/kwork-parser/mitmproxy/export.har
```
---
_Updated: 2026-03-22_

46
mitmproxy/auth.py
View File

@ -1,46 +0,0 @@
#!/usr/bin/env python3
"""
mitmproxy addon: HTTP Basic Auth
Usage: mitmdump -s auth.py --set auth_user=admin --set auth_pass=secret
"""
from mitmproxy import http, ctx
class HTTPAuth:
def __init__(self):
self.auth_user = ctx.options.auth_user
self.auth_pass = ctx.options.auth_pass
def request(self, flow: http.HTTPFlow):
# Skip auth for mitm.it (certificate download)
if "mitm.it" in flow.request.pretty_host:
return
# Check for Authorization header
auth_header = flow.request.headers.get("Authorization", "")
if not auth_header.startswith("Basic "):
self.challenge_auth(flow)
return
# Decode and verify credentials
import base64
try:
credentials = base64.b64decode(auth_header[6:]).decode()
username, password = credentials.split(":", 1)
if username != self.auth_user or password != self.auth_pass:
self.challenge_auth(flow)
except Exception:
self.challenge_auth(flow)
def challenge_auth(self, flow: http.HTTPFlow):
flow.response = http.Response.make(
407,
b"Proxy Authentication Required",
{"Proxy-Authenticate": 'Basic realm="mitmproxy"'}
)
addons = [HTTPAuth()]

34
mitmproxy/mitmproxy.conf
View File

@ -1,34 +0,0 @@
# mitmproxy configuration
# Authentication
auth_user = "claw"
auth_pass = "KworkParser2026!"
# Network
mode = "regular"
listen_port = 8080
listen_host = "0.0.0.0"
# SSL
ssl_insecure = false
add_upstream_certs_to_client_chain = true
# Logging
verbose = true
termlog_verbosity = "info"
# Save flows
save_stream_file = "/root/kwork-parser/mitmproxy/flows.mitm"
# Ignore common CDNs and analytics (reduce noise)
ignore_hosts = [
"~u cdn.*\\.google\\.com",
"~u .*\\.google-analytics\\.com",
"~u .*\\.yandex\\.ru",
"~u .*\\.yandex\\.net",
"~u .*\\.facebook\\.com",
]
# Keep only Kwork and related domains
# Uncomment to filter strictly:
# filter = "~u (kwork\\.ru|kworks\\.ru|kwork-api\\.ru)"

72
mitmproxy/start.sh
View File

@ -1,72 +0,0 @@
#!/bin/bash
# Start mitmproxy with authentication
set -e
CONFIG_DIR="/root/kwork-parser/mitmproxy"
LOG_FILE="/root/kwork-parser/mitmproxy/mitmproxy.log"
PID_FILE="/root/kwork-parser/mitmproxy/mitmproxy.pid"
echo "=== Mitmproxy Startup ==="
echo "Config: $CONFIG_DIR/mitmproxy.conf"
echo "Auth addon: $CONFIG_DIR/auth.py"
echo "Log: $LOG_FILE"
echo ""
# Check if already running
if [ -f "$PID_FILE" ]; then
OLD_PID=$(cat "$PID_FILE")
if kill -0 "$OLD_PID" 2>/dev/null; then
echo "⚠️ mitmproxy already running (PID: $OLD_PID)"
echo "Stop with: sudo systemctl stop mitmproxy"
exit 1
fi
fi
# Create certificate if not exists
if [ ! -f ~/.mitmproxy/mitmproxy-ca.pem ]; then
echo "📜 Generating SSL certificate..."
mitmproxy --version > /dev/null
fi
# Start mitmproxy
echo "🚀 Starting mitmproxy..."
nohup mitmdump \
-s "$CONFIG_DIR/auth.py" \
--set auth_user=claw \
--set auth_pass="KworkParser2026!" \
-p 8080 \
--mode regular \
--set ssl_insecure=false \
--set add_upstream_certs_to_client_chain=true \
-w "$CONFIG_DIR/flows.mitm" \
-v \
> "$LOG_FILE" 2>&1 &
echo $! > "$PID_FILE"
sleep 2
# Check if running
if kill -0 $(cat "$PID_FILE") 2>/dev/null; then
echo "✅ mitmproxy started successfully!"
echo ""
echo "📱 Phone configuration:"
echo " Proxy: $(curl -s ifconfig.me):8080"
echo " Login: claw"
echo " Password: KworkParser2026!"
echo ""
echo "🔒 Certificate:"
echo " http://$(curl -s ifconfig.me):8080/mitmproxy/mitmproxy-ca.cer"
echo ""
echo "📊 View logs:"
echo " tail -f $LOG_FILE"
echo " python3 /root/kwork-parser/scripts/view_flows.py"
echo ""
echo "🛑 Stop:"
echo " sudo systemctl stop mitmproxy"
echo " or: kill $(cat $PID_FILE)"
else
echo "❌ Failed to start mitmproxy"
echo "Check log: $LOG_FILE"
exit 1
fi

97
scripts/mitmproxy-to-har.py
View File

@ -1,97 +0,0 @@
#!/usr/bin/env python3
"""
Convert mitmproxy flow dump to HAR format
Usage: mitmproxy-to-har.py <input.mitm> <output.har>
"""
import json
import sys
from datetime import datetime
from mitmproxy import io
from mitmproxy.http import HTTPFlow
def flow_to_har_entry(flow: HTTPFlow) -> dict:
"""Convert single flow to HAR entry."""
if not flow.request or not flow.response:
return None
entry = {
"startedDateTime": datetime.fromtimestamp(flow.request.timestamp_start).isoformat() + "Z",
"time": int((flow.response.timestamp_end - flow.request.timestamp_start) * 1000) if flow.response.timestamp_end else 0,
"request": {
"method": flow.request.method,
"url": flow.request.url,
"httpVersion": flow.request.http_version,
"headers": [{"name": k, "value": v} for k, v in flow.request.headers.items()],
"queryString": [{"name": k, "value": v} for k, v in flow.request.query.items()],
"cookies": [{"name": k, "value": v} for k, v in flow.request.cookies.items()],
"headersSize": -1,
"bodySize": len(flow.request.content) if flow.request.content else 0,
},
"response": {
"status": flow.response.status_code,
"statusText": flow.response.reason,
"httpVersion": flow.response.http_version,
"headers": [{"name": k, "value": v} for k, v in flow.response.headers.items()],
"cookies": [{"name": k, "value": v} for k, v in flow.response.cookies.items()],
"content": {
"size": len(flow.response.content) if flow.response.content else 0,
"mimeType": flow.response.headers.get("Content-Type", "unknown").split(";")[0],
"text": flow.response.get_text() if flow.response.content else ""
},
"redirectURL": flow.response.headers.get("Location", ""),
"headersSize": -1,
"bodySize": len(flow.response.content) if flow.response.content else 0,
},
"cache": {},
"timings": {
"send": 0,
"wait": int((flow.response.timestamp_start - flow.request.timestamp_end) * 1000) if flow.response.timestamp_start and flow.request.timestamp_end else 0,
"receive": int((flow.response.timestamp_end - flow.response.timestamp_start) * 1000) if flow.response.timestamp_end and flow.response.timestamp_start else 0,
}
}
# Add POST data if present
if flow.request.method == "POST" and flow.request.content:
entry["request"]["postData"] = {
"mimeType": flow.request.headers.get("Content-Type", "application/octet-stream"),
"text": flow.request.get_text() if flow.request.content else ""
}
return entry
def convert_mitm_to_har(input_file: str, output_file: str):
"""Convert mitmproxy dump to HAR."""
har = {
"log": {
"version": "1.2",
"creator": {
"name": "mitmproxy-to-har",
"version": "1.0"
},
"entries": []
}
}
with open(input_file, 'rb') as f:
reader = io.FlowReader(f)
for flow in reader.read():
if isinstance(flow, HTTPFlow):
entry = flow_to_har_entry(flow)
if entry:
har["log"]["entries"].append(entry)
with open(output_file, 'w', encoding='utf-8') as f:
json.dump(har, f, indent=2, ensure_ascii=False)
print(f"Converted {len(har['log']['entries'])} flows to {output_file}")
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: mitmproxy-to-har.py <input.mitm> <output.har>")
sys.exit(1)
convert_mitm_to_har(sys.argv[1], sys.argv[2])

106
scripts/view_flows.py
View File

@ -1,106 +0,0 @@
#!/usr/bin/env python3
"""
View mitmproxy flows from log file
Usage: python3 view_flows.py [options]
Options:
--last N Show last N requests (default: 20)
--domain X Filter by domain
--method X Filter by method (GET/POST)
--json Show only JSON responses
--full Show full request/response
"""
import sys
import json
from pathlib import Path
from mitmproxy import io
FLOWS_FILE = "/root/kwork-parser/mitmproxy/flows.mitm"
def view_flows(last_n=20, domain_filter=None, method_filter=None, json_only=False, full=False):
"""View flows from mitmproxy dump."""
if not Path(FLOWS_FILE).exists():
print(f"❌ Flows file not found: {FLOWS_FILE}")
print("Make sure mitmproxy is running and capturing traffic.")
return
flows = []
with open(FLOWS_FILE, 'rb') as f:
reader = io.FlowReader(f)
for flow in reader.read():
from mitmproxy.http import HTTPFlow
if isinstance(flow, HTTPFlow) and flow.request and flow.response:
flows.append(flow)
# Filter
if domain_filter:
flows = [f for f in flows if domain_filter in f.request.url]
if method_filter:
flows = [f for f in flows if f.request.method == method_filter]
if json_only:
flows = [f for f in flows if f.response.headers.get("Content-Type", "").startswith("application/json")]
# Sort by time (newest first)
flows.sort(key=lambda x: x.request.timestamp_start, reverse=True)
# Limit
flows = flows[:last_n]
# Display
print(f"📊 Showing {len(flows)} flows (newest first)\n")
for i, flow in enumerate(flows, 1):
method = flow.request.method
url = flow.request.url
status = flow.response.status_code
size = len(flow.response.content) if flow.response.content else 0
content_type = flow.response.headers.get("Content-Type", "unknown").split(";")[0]
time_ms = int((flow.response.timestamp_end - flow.request.timestamp_start) * 1000) if flow.response.timestamp_end else 0
# Color coding
status_color = "🟢" if status < 400 else "🟡" if status < 500 else "🔴"
print(f"{i}. {status_color} {method} {url[:80]}")
print(f" Status: {status} | Size: {size:,} bytes | Time: {time_ms}ms | Type: {content_type}")
if full:
print(f"\n Request Headers:")
for k, v in flow.request.headers.items():
print(f" {k}: {v[:100]}")
if flow.request.method == "POST" and flow.request.content:
print(f"\n Request Body:")
print(f" {flow.request.get_text()[:500]}")
print(f"\n Response:")
print(f" {flow.response.get_text()[:500]}")
print()
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="View mitmproxy flows")
parser.add_argument("--last", type=int, default=20, help="Show last N requests")
parser.add_argument("--domain", type=str, help="Filter by domain")
parser.add_argument("--method", type=str, help="Filter by method")
parser.add_argument("--json", action="store_true", help="Show only JSON responses")
parser.add_argument("--full", action="store_true", help="Show full request/response")
args = parser.parse_args()
view_flows(
last_n=args.last,
domain_filter=args.domain,
method_filter=args.method,
json_only=args.json,
full=args.full
)