Http
the_data_packet.utils.http
¶
HTTP client utility.
logger = get_logger(__name__)
module-attribute
¶
HTTPClient
¶
Simple HTTP client with error handling and configuration.
timeout = timeout or config.http_timeout
instance-attribute
¶
user_agent = user_agent or config.user_agent
instance-attribute
¶
session = requests.Session()
instance-attribute
¶
__init__(timeout: Optional[int] = None, user_agent: Optional[str] = None)
¶
Initialize HTTP client.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout
|
Optional[int]
|
Request timeout (defaults to config) |
None
|
user_agent
|
Optional[str]
|
User agent string (defaults to config) |
None
|
get(url: str, **kwargs: Any) -> requests.Response
¶
Make a GET request.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
URL to fetch |
required |
**kwargs
|
Any
|
Additional arguments passed to requests.get |
{}
|
Returns:
| Type | Description |
|---|---|
Response
|
Response object |
Raises:
| Type | Description |
|---|---|
NetworkError
|
If request fails |
get_soup(url: str, **kwargs: Any) -> BeautifulSoup
¶
Get a URL and return parsed HTML.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
URL to fetch |
required |
**kwargs
|
Any
|
Additional arguments passed to get() |
{}
|
Returns:
| Type | Description |
|---|---|
BeautifulSoup
|
BeautifulSoup object |
Raises:
| Type | Description |
|---|---|
NetworkError
|
If request fails |